mbutrovich commented on code in PR #334:
URL: https://github.com/apache/datafusion-spark/pull/334#discussion_r4602747692


##########
LICENSE.txt:
##########
@@ -200,3 +200,15 @@
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
+
+--------------------------------------------------------------------------------
+
+This product includes code translated from fdlibm 5.3 (Freely Distributable
+LIBM), in crates/spark/src/function/math/fdlibm.rs, under the following terms:
+
+  Copyright (C) 1993 by Sun Microsystems, Inc. All rights reserved.
+
+  Developed at SunSoft, a Sun Microsystems, Inc. business.
+  Permission to use, copy, modify, and distribute this
+  software is freely granted, provided that this notice
+  is preserved.

Review Comment:
   > Reviewers, please confirm this is acceptable under ASF policy.
   
   The fdlibm notice is the SPDX 
[`SunPro`](https://spdx.org/licenses/SunPro.html) license, which isn't named in 
the ASF's [Category A 
list](https://www.apache.org/legal/resolved.html#category-a). There is 
precedent: Lucene ships fdlibm-derived code with the Apache header on top and 
the Sun notice in a comment 
([`SloppyMath.java`](https://github.com/apache/lucene/blob/596ef11ddf96217f519880fef8153a5e7cdc4848/lucene/core/src/java/org/apache/lucene/util/SloppyMath.java#L1-L29)).
 The ASF header policy also says [not to put the Apache header on third-party 
source 
files](https://www.apache.org/legal/src-headers.html#3party-noapachelicense), 
and it leaves [major modifications to the 
PMC](https://www.apache.org/legal/src-headers.html#3party-majormodpmc). A 
line-by-line translation falls between those two cases.
   
   Could you ask on legal-discuss@ whether `SunPro` is Category A and which 
header `fdlibm.rs` should carry, and link the answer here? I'd like that 
settled before this merges.



##########
crates/sqllogictest/test_files/math/log.slt:
##########
@@ -8,7 +8,102 @@
 # For more information, please see:
 #   https://github.com/apache/datafusion/issues/15914
 
+
 ## Original Query: SELECT log(10, 100);
 ## PySpark 3.5.5 Result: {'LOG(10, 100)': 2.0, 'typeof(LOG(10, 100))': 
'double', 'typeof(10)': 'int', 'typeof(100)': 'int'}
-#query
-#SELECT log(10::int, 100::int);
+query R
+SELECT log(10::int, 100::int);
+----
+2
+
+query T
+SELECT arrow_typeof(log(10::int, 100::int));
+----
+Float64
+
+query RR
+SELECT log(2::bigint, 1024::bigint), log(0.5::double, 8::double);
+----
+10 -3
+
+# Computed as ln(expr) / ln(base), so results can be off in the last digit.
+# Rust's f64::ln gives 4 for log(3, 81); StrictMath.log, which Spark uses,
+# gives 4.000000000000001.
+query RR
+SELECT log(10::int, 1000::int), log(3::int, 81::int);
+----
+2.9999999999999996 4.000000000000001
+
+query R
+SELECT log(10::decimal(5, 1), 100::decimal(5, 1));
+----
+2
+
+# One argument is the natural logarithm, not base 10 as in DataFusion
+query RRR
+SELECT log(1::int), log(2::double), log(10::decimal(5, 2));
+----
+0 0.6931471805599453 2.302585092994046
+
+query T
+SELECT arrow_typeof(log(1::int));
+----
+Float64
+
+# NULL if either argument is <= 0
+query RRRRRR
+SELECT log(0::int, 10::int), log(-2::int, 10::int), log(2::int, 0::int), 
log(2::int, -8::int), log(-1::double), log(0::double);
+----
+NULL NULL NULL NULL NULL NULL
+
+# ...even when the other argument is NaN
+query RR
+SELECT log(0::double, 'NaN'::double), log('NaN'::double, -1::double);
+----
+NULL NULL
+
+query RRRR
+SELECT log(NULL::double, 2::double), log(2::double, NULL::double), 
log(NULL::double), log(NULL::double, NULL::double);
+----
+NULL NULL NULL NULL
+
+query RRR
+SELECT log('NaN'::double, 2::double), log(2::double, 'NaN'::double), 
log('NaN'::double);
+----
+NaN NaN NaN
+
+# A base of 1 divides by ln(1) = 0
+query RRR
+SELECT log(1::int, 2::int), log(1::double, 0.5::double), log(1::int, 1::int);
+----
+Infinity -Infinity NaN
+
+query RRR
+SELECT log('Infinity'::double, 2::double), log(2::double, 'Infinity'::double), 
log('Infinity'::double, 'Infinity'::double);
+----
+0 Infinity NaN
+
+# Array path, including NULL and non-positive rows
+query R
+SELECT log(b, x) FROM (VALUES (10::double, 100::double), (2::double, 
0::double), (NULL::double, 3::double), (0::double, 3::double), (2::double, 
8::double)) AS t(b, x);
+----
+2
+NULL
+NULL
+NULL
+3
+
+# Scalar base with an array expr
+query R
+SELECT log(2::int, x) FROM (VALUES (8::double), (-1::double), (NULL::double)) 
AS t(x);
+----
+3
+NULL
+NULL

Review Comment:
   The file covers a scalar base with an array `expr`, but not an array base 
with a scalar `expr`. Could we add that case?
   
   ```suggestion
   # Scalar base with an array expr
   query R
   SELECT log(2::int, x) FROM (VALUES (8::double), (-1::double), 
(NULL::double)) AS t(x);
   ----
   3
   NULL
   NULL
   
   # Array base with a scalar expr
   query R
   SELECT log(b, 8::double) FROM (VALUES (2::double), (0::double), 
(NULL::double)) AS t(b);
   ----
   3
   NULL
   NULL
   ```



##########
crates/spark/src/function/math/log.rs:
##########
@@ -0,0 +1,317 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for 
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) 
under one or more contributor license agreements
+
+//! Spark-compatible logarithm functions: `ln`, `log10`, `log2`, `log1p` and 
`log`.
+//!
+//! Spark follows Hive here: an input at or below the function's asymptote
+//! (0 for `ln`, `log10`, `log2` and both arguments of `log`, -1 for `log1p`)
+//! returns NULL instead of the NaN or -Infinity that IEEE 754 gives. NaN is
+//! not below the asymptote, so a NaN input returns NaN.
+//!
+//! DataFusion's built-ins return -Infinity or NaN for those inputs, and its
+//! one-argument `log` is base 10 where Spark's is the natural logarithm.
+//!
+//! Spark computes the logarithms with `java.lang.StrictMath`, so these use the
+//! fdlibm translation in [`super::fdlibm`] to return the same bits. Rust's
+//! `f64::ln` differs in the last digit for some inputs.
+
+use std::sync::Arc;
+
+use arrow::array::{ArrayRef, AsArray, Float64Array};
+use arrow::datatypes::{DataType, Float64Type};
+use datafusion_common::utils::take_function_args;
+use datafusion_common::{Result, exec_err};
+use datafusion_expr::{
+    ColumnarValue, Documentation, ScalarFunctionArgs, ScalarUDFImpl, Signature,
+    TypeSignature, Volatility,
+};
+use datafusion_functions::utils::make_scalar_function;
+use datafusion_macros::user_doc;
+
+use super::fdlibm;
+
+/// Applies `f` to every value of a Float64 array, returning NULL where the
+/// value is at or below `asymptote`, as Spark's `UnaryLogExpression` does.
+fn unary_log(
+    name: &str,
+    args: &[ArrayRef],
+    asymptote: f64,
+    f: impl Fn(f64) -> f64,
+) -> Result<ArrayRef> {
+    let [arg] = take_function_args(name, args)?;
+    let result: Float64Array = arg
+        .as_primitive::<Float64Type>()
+        .unary_opt(|x| if x <= asymptote { None } else { Some(f(x)) });
+    Ok(Arc::new(result))
+}
+
+/// Defines a one-argument Spark logarithm UDF over DOUBLE.
+macro_rules! unary_log_udf {
+    ($struct:ident, $name:literal, $asymptote:expr, $f:expr) => {
+        impl Default for $struct {
+            fn default() -> Self {
+                Self::new()
+            }
+        }
+
+        impl $struct {
+            pub fn new() -> Self {
+                Self {
+                    signature: Signature::exact(
+                        vec![DataType::Float64],
+                        Volatility::Immutable,
+                    ),
+                }
+            }
+        }
+
+        impl ScalarUDFImpl for $struct {
+            fn name(&self) -> &str {
+                $name
+            }
+
+            fn signature(&self) -> &Signature {
+                &self.signature
+            }
+
+            fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> 
{
+                Ok(DataType::Float64)
+            }
+
+            fn documentation(&self) -> Option<&Documentation> {
+                self.doc()
+            }
+
+            fn invoke_with_args(
+                &self,
+                args: ScalarFunctionArgs,
+            ) -> Result<ColumnarValue> {
+                make_scalar_function(
+                    |args: &[ArrayRef]| unary_log($name, args, $asymptote, $f),
+                    vec![],
+                )(&args.args)
+            }
+        }
+    };
+}
+
+/// Spark-compatible `ln` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#ln>
+#[user_doc(
+    doc_section(label = "Math Functions"),
+    description = "Returns the natural logarithm (base e) of `expr` as a 
DOUBLE. The argument is \
+        cast to DOUBLE. Unlike DataFusion's built-in `ln`, which returns 
-Infinity for 0 and NaN for \
+        a negative number, an input less than or equal to 0 returns NULL. 
Returns NULL if the input \
+        is NULL, and NaN if it is NaN.",
+    syntax_example = "ln(expr)",
+    sql_example = r#"```sql
+> SELECT ln(1);
+0
+> SELECT ln(0);
+NULL
+```"#,
+    argument(
+        name = "expr",
+        description = "Numeric expression, cast to DOUBLE. NULL, 0 or a 
negative number returns NULL."
+    )
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLn {
+    signature: Signature,
+}
+
+unary_log_udf!(SparkLn, "ln", 0.0, fdlibm::log);
+
+/// Spark-compatible `log10` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#log10>
+#[user_doc(
+    doc_section(label = "Math Functions"),
+    description = "Returns the base 10 logarithm of `expr` as a DOUBLE. The 
argument is cast to \
+        DOUBLE. Unlike DataFusion's built-in `log10`, which returns -Infinity 
for 0 and NaN for a \
+        negative number, an input less than or equal to 0 returns NULL. 
Returns NULL if the input \
+        is NULL, and NaN if it is NaN.",
+    syntax_example = "log10(expr)",
+    sql_example = r#"```sql
+> SELECT log10(10);
+1
+> SELECT log10(-1);
+NULL
+```"#,
+    argument(
+        name = "expr",
+        description = "Numeric expression, cast to DOUBLE. NULL, 0 or a 
negative number returns NULL."
+    )
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLog10 {
+    signature: Signature,
+}
+
+unary_log_udf!(SparkLog10, "log10", 0.0, fdlibm::log10);
+
+/// Spark-compatible `log2` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#log2>
+#[user_doc(
+    doc_section(label = "Math Functions"),
+    description = "Returns the base 2 logarithm of `expr` as a DOUBLE. The 
argument is cast to \
+        DOUBLE. Like Spark, it is computed as `ln(expr) / ln(2)`, so it can 
differ from an exact \
+        base 2 logarithm in the last digit. Unlike DataFusion's built-in 
`log2`, which returns \
+        -Infinity for 0 and NaN for a negative number, an input less than or 
equal to 0 returns \
+        NULL. Returns NULL if the input is NULL, and NaN if it is NaN.",
+    syntax_example = "log2(expr)",
+    sql_example = r#"```sql
+> SELECT log2(2);
+1
+> SELECT log2(0);
+NULL
+```"#,
+    argument(
+        name = "expr",
+        description = "Numeric expression, cast to DOUBLE. NULL, 0 or a 
negative number returns NULL."
+    )
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLog2 {
+    signature: Signature,
+}
+
+// Spark computes `StrictMath.log(x) / StrictMath.log(2)` rather than a true
+// base 2 logarithm, which differs in the last bit for some inputs.
+unary_log_udf!(SparkLog2, "log2", 0.0, |x: f64| {
+    fdlibm::log(x) / fdlibm::log(2.0)
+});
+
+/// Spark-compatible `log1p` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#log1p>
+#[user_doc(
+    doc_section(label = "Math Functions"),
+    description = "Returns `ln(1 + expr)` as a DOUBLE, computed without the 
precision loss of \
+        adding 1 to `expr` when `expr` is close to 0. DataFusion has no 
built-in `log1p`. The \
+        argument is cast to DOUBLE. An input less than or equal to -1 returns 
NULL. Returns NULL \
+        if the input is NULL, and NaN if it is NaN.",
+    syntax_example = "log1p(expr)",
+    sql_example = r#"```sql
+> SELECT log1p(0);
+0
+> SELECT log1p(-1);
+NULL
+```"#,
+    argument(
+        name = "expr",
+        description = "Numeric expression, cast to DOUBLE. NULL or a number 
less than or equal to -1 returns NULL."
+    )
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLog1p {
+    signature: Signature,
+}
+
+unary_log_udf!(SparkLog1p, "log1p", -1.0, fdlibm::log1p);
+
+/// Spark-compatible `log` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#log>
+#[user_doc(
+    doc_section(label = "Math Functions"),
+    description = "Returns the logarithm of `expr` in base `base` as a DOUBLE, 
computed as \
+        `ln(expr) / ln(base)`. With one argument it returns the natural 
logarithm of `expr`, \
+        where DataFusion's built-in `log` returns the base 10 logarithm. Both 
arguments are cast \
+        to DOUBLE. Returns NULL if either argument is NULL or less than or 
equal to 0, otherwise \
+        NaN if either is NaN. A base of 1 divides by 0, so it returns 
Infinity, -Infinity, or NaN \
+        when `expr` is also 1.",
+    syntax_example = "log([base, ]expr)",
+    sql_example = r#"```sql
+> SELECT log(10, 100);
+2
+> SELECT log(1);
+0
+> SELECT log(0, 10);
+NULL
+```"#,
+    argument(
+        name = "base",
+        description = "Optional numeric base, cast to DOUBLE. Defaults to e. 
NULL, 0 or a negative number returns NULL."
+    ),
+    argument(
+        name = "expr",
+        description = "Numeric expression, cast to DOUBLE. NULL, 0 or a 
negative number returns NULL."
+    )
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLog {
+    signature: Signature,
+}
+
+impl Default for SparkLog {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+impl SparkLog {
+    pub fn new() -> Self {
+        Self {
+            signature: Signature::one_of(
+                vec![
+                    TypeSignature::Exact(vec![DataType::Float64]),
+                    TypeSignature::Exact(vec![DataType::Float64, 
DataType::Float64]),
+                ],
+                Volatility::Immutable,
+            ),
+        }
+    }
+}
+
+impl ScalarUDFImpl for SparkLog {
+    fn name(&self) -> &str {
+        "log"
+    }
+
+    fn signature(&self) -> &Signature {
+        &self.signature
+    }
+
+    fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
+        Ok(DataType::Float64)
+    }
+
+    fn documentation(&self) -> Option<&Documentation> {
+        self.doc()
+    }
+
+    fn invoke_with_args(&self, args: ScalarFunctionArgs) -> 
Result<ColumnarValue> {
+        make_scalar_function(spark_log, vec![])(&args.args)
+    }
+}
+
+fn spark_log(args: &[ArrayRef]) -> Result<ArrayRef> {
+    match args {
+        // Spark's one-argument `log` is `Logarithm(EulerNumber(), expr)`,
+        // which its generated code evaluates as `StrictMath.log(expr)`.
+        [_] => unary_log("log", args, 0.0, fdlibm::log),
+        [base, value] => {
+            let base = base.as_primitive::<Float64Type>();
+            let value = value.as_primitive::<Float64Type>();
+            let result: Float64Array = base
+                .iter()
+                .zip(value.iter())
+                .map(|(base, value)| match (base, value) {
+                    // NaN is not <= 0, so a NaN argument yields NaN
+                    // unless the other argument is <= 0.
+                    (Some(base), Some(value)) if base <= 0.0 || value <= 0.0 
=> None,
+                    (Some(base), Some(value)) => {
+                        Some(fdlibm::log(value) / fdlibm::log(base))
+                    }
+                    _ => None,
+                })
+                .collect();
+            Ok(Arc::new(result))

Review Comment:
   With a literal base, which is how `log` is usually called, 
`make_scalar_function` expands the base into an array, so `fdlibm::log(base)` 
runs on the same value for every row. Could `invoke_with_args` handle a scalar 
base before `make_scalar_function`? It would compute `fdlibm::log(base)` once, 
map `value` with `unary_opt`, and return all NULLs when the base is NULL or `<= 
0`.



##########
crates/spark/src/function/math/log.rs:
##########
@@ -0,0 +1,317 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for 
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) 
under one or more contributor license agreements
+
+//! Spark-compatible logarithm functions: `ln`, `log10`, `log2`, `log1p` and 
`log`.
+//!
+//! Spark follows Hive here: an input at or below the function's asymptote
+//! (0 for `ln`, `log10`, `log2` and both arguments of `log`, -1 for `log1p`)
+//! returns NULL instead of the NaN or -Infinity that IEEE 754 gives. NaN is
+//! not below the asymptote, so a NaN input returns NaN.
+//!
+//! DataFusion's built-ins return -Infinity or NaN for those inputs, and its
+//! one-argument `log` is base 10 where Spark's is the natural logarithm.
+//!
+//! Spark computes the logarithms with `java.lang.StrictMath`, so these use the
+//! fdlibm translation in [`super::fdlibm`] to return the same bits. Rust's
+//! `f64::ln` differs in the last digit for some inputs.
+
+use std::sync::Arc;
+
+use arrow::array::{ArrayRef, AsArray, Float64Array};
+use arrow::datatypes::{DataType, Float64Type};
+use datafusion_common::utils::take_function_args;
+use datafusion_common::{Result, exec_err};
+use datafusion_expr::{
+    ColumnarValue, Documentation, ScalarFunctionArgs, ScalarUDFImpl, Signature,
+    TypeSignature, Volatility,
+};
+use datafusion_functions::utils::make_scalar_function;
+use datafusion_macros::user_doc;
+
+use super::fdlibm;
+
+/// Applies `f` to every value of a Float64 array, returning NULL where the
+/// value is at or below `asymptote`, as Spark's `UnaryLogExpression` does.
+fn unary_log(
+    name: &str,
+    args: &[ArrayRef],
+    asymptote: f64,
+    f: impl Fn(f64) -> f64,
+) -> Result<ArrayRef> {
+    let [arg] = take_function_args(name, args)?;
+    let result: Float64Array = arg
+        .as_primitive::<Float64Type>()
+        .unary_opt(|x| if x <= asymptote { None } else { Some(f(x)) });
+    Ok(Arc::new(result))
+}
+
+/// Defines a one-argument Spark logarithm UDF over DOUBLE.
+macro_rules! unary_log_udf {
+    ($struct:ident, $name:literal, $asymptote:expr, $f:expr) => {
+        impl Default for $struct {
+            fn default() -> Self {
+                Self::new()
+            }
+        }
+
+        impl $struct {
+            pub fn new() -> Self {
+                Self {
+                    signature: Signature::exact(
+                        vec![DataType::Float64],
+                        Volatility::Immutable,
+                    ),

Review Comment:
   Spark implicitly casts a string argument to DOUBLE here 
([`UnaryMathExpression`](https://github.com/apache/spark/blob/32f7299601108917fb01920a54e084595b7b3bf8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala#L66-L70)
 has `ImplicitCastInputTypes` with `Seq(DoubleType)`). So `SELECT ln('10')` 
returns `2.302585092994046` in Spark 4.2.0 with ANSI mode on or off, and 
`log('2', '8')` returns `3.0`. Here both fail at planning with `coercion from 
Utf8 to the signature Exact(Float64) failed`. #266 also lists STRING arguments 
among the cases to test.
   
   Could the signatures accept strings the way 
[`date_from_unix_date.rs`](https://github.com/apache/datafusion-spark/blob/125600164aa825e2fe805887dfdb384d4182f2a8/crates/spark/src/function/datetime/date_from_unix_date.rs#L59-L72)
 does, with `Coercion::new_implicit` and 
`TypeSignatureClass::Native(logical_string())`? With ANSI mode off, a malformed 
string returns NULL in Spark but errors here. That case could be a 
commented-out query citing #88, as in 
[`date_from_unix_date.slt`](https://github.com/apache/datafusion-spark/blob/125600164aa825e2fe805887dfdb384d4182f2a8/crates/sqllogictest/test_files/datetime/date_from_unix_date.slt?plain=1#L103-L107).



##########
crates/spark/src/function/math/log.rs:
##########
@@ -0,0 +1,317 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for 
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) 
under one or more contributor license agreements
+
+//! Spark-compatible logarithm functions: `ln`, `log10`, `log2`, `log1p` and 
`log`.
+//!
+//! Spark follows Hive here: an input at or below the function's asymptote
+//! (0 for `ln`, `log10`, `log2` and both arguments of `log`, -1 for `log1p`)
+//! returns NULL instead of the NaN or -Infinity that IEEE 754 gives. NaN is
+//! not below the asymptote, so a NaN input returns NaN.
+//!
+//! DataFusion's built-ins return -Infinity or NaN for those inputs, and its
+//! one-argument `log` is base 10 where Spark's is the natural logarithm.
+//!
+//! Spark computes the logarithms with `java.lang.StrictMath`, so these use the
+//! fdlibm translation in [`super::fdlibm`] to return the same bits. Rust's
+//! `f64::ln` differs in the last digit for some inputs.
+
+use std::sync::Arc;
+
+use arrow::array::{ArrayRef, AsArray, Float64Array};
+use arrow::datatypes::{DataType, Float64Type};
+use datafusion_common::utils::take_function_args;
+use datafusion_common::{Result, exec_err};
+use datafusion_expr::{
+    ColumnarValue, Documentation, ScalarFunctionArgs, ScalarUDFImpl, Signature,
+    TypeSignature, Volatility,
+};
+use datafusion_functions::utils::make_scalar_function;
+use datafusion_macros::user_doc;
+
+use super::fdlibm;
+
+/// Applies `f` to every value of a Float64 array, returning NULL where the
+/// value is at or below `asymptote`, as Spark's `UnaryLogExpression` does.
+fn unary_log(
+    name: &str,
+    args: &[ArrayRef],
+    asymptote: f64,
+    f: impl Fn(f64) -> f64,
+) -> Result<ArrayRef> {
+    let [arg] = take_function_args(name, args)?;
+    let result: Float64Array = arg
+        .as_primitive::<Float64Type>()

Review Comment:
   `as_primitive` panics if the array isn't `Float64`. SQL always coerces 
first, but a caller that builds the physical expression directly, as Comet 
does, can pass another type. With an `Int32` array, 
`SparkLn::new().invoke_with_args(...)` panics with `primitive array` 
(arrow-array 60.1.0 `cast.rs:898`). `spark_log` has the same downcasts at lines 
298-299. Could these return an error instead, the way 
[`expm1.rs`](https://github.com/apache/datafusion-spark/blob/125600164aa825e2fe805887dfdb384d4182f2a8/crates/spark/src/function/math/expm1.rs#L80-L93)
 does with `unsupported_data_type_exec_err`?



##########
crates/spark/src/function/math/log.rs:
##########
@@ -0,0 +1,317 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for 
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) 
under one or more contributor license agreements
+
+//! Spark-compatible logarithm functions: `ln`, `log10`, `log2`, `log1p` and 
`log`.
+//!
+//! Spark follows Hive here: an input at or below the function's asymptote
+//! (0 for `ln`, `log10`, `log2` and both arguments of `log`, -1 for `log1p`)
+//! returns NULL instead of the NaN or -Infinity that IEEE 754 gives. NaN is
+//! not below the asymptote, so a NaN input returns NaN.
+//!
+//! DataFusion's built-ins return -Infinity or NaN for those inputs, and its
+//! one-argument `log` is base 10 where Spark's is the natural logarithm.
+//!
+//! Spark computes the logarithms with `java.lang.StrictMath`, so these use the
+//! fdlibm translation in [`super::fdlibm`] to return the same bits. Rust's
+//! `f64::ln` differs in the last digit for some inputs.
+
+use std::sync::Arc;
+
+use arrow::array::{ArrayRef, AsArray, Float64Array};
+use arrow::datatypes::{DataType, Float64Type};
+use datafusion_common::utils::take_function_args;
+use datafusion_common::{Result, exec_err};
+use datafusion_expr::{
+    ColumnarValue, Documentation, ScalarFunctionArgs, ScalarUDFImpl, Signature,
+    TypeSignature, Volatility,
+};
+use datafusion_functions::utils::make_scalar_function;
+use datafusion_macros::user_doc;
+
+use super::fdlibm;
+
+/// Applies `f` to every value of a Float64 array, returning NULL where the
+/// value is at or below `asymptote`, as Spark's `UnaryLogExpression` does.
+fn unary_log(
+    name: &str,
+    args: &[ArrayRef],
+    asymptote: f64,
+    f: impl Fn(f64) -> f64,
+) -> Result<ArrayRef> {
+    let [arg] = take_function_args(name, args)?;
+    let result: Float64Array = arg
+        .as_primitive::<Float64Type>()
+        .unary_opt(|x| if x <= asymptote { None } else { Some(f(x)) });
+    Ok(Arc::new(result))
+}
+
+/// Defines a one-argument Spark logarithm UDF over DOUBLE.
+macro_rules! unary_log_udf {
+    ($struct:ident, $name:literal, $asymptote:expr, $f:expr) => {
+        impl Default for $struct {
+            fn default() -> Self {
+                Self::new()
+            }
+        }
+
+        impl $struct {
+            pub fn new() -> Self {
+                Self {
+                    signature: Signature::exact(
+                        vec![DataType::Float64],
+                        Volatility::Immutable,
+                    ),
+                }
+            }
+        }
+
+        impl ScalarUDFImpl for $struct {
+            fn name(&self) -> &str {
+                $name
+            }
+
+            fn signature(&self) -> &Signature {
+                &self.signature
+            }
+
+            fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> 
{
+                Ok(DataType::Float64)
+            }
+
+            fn documentation(&self) -> Option<&Documentation> {
+                self.doc()
+            }
+
+            fn invoke_with_args(
+                &self,
+                args: ScalarFunctionArgs,
+            ) -> Result<ColumnarValue> {
+                make_scalar_function(
+                    |args: &[ArrayRef]| unary_log($name, args, $asymptote, $f),
+                    vec![],
+                )(&args.args)
+            }
+        }
+    };
+}
+
+/// Spark-compatible `ln` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#ln>
+#[user_doc(
+    doc_section(label = "Math Functions"),
+    description = "Returns the natural logarithm (base e) of `expr` as a 
DOUBLE. The argument is \
+        cast to DOUBLE. Unlike DataFusion's built-in `ln`, which returns 
-Infinity for 0 and NaN for \
+        a negative number, an input less than or equal to 0 returns NULL. 
Returns NULL if the input \
+        is NULL, and NaN if it is NaN.",
+    syntax_example = "ln(expr)",
+    sql_example = r#"```sql
+> SELECT ln(1);
+0
+> SELECT ln(0);
+NULL
+```"#,
+    argument(
+        name = "expr",
+        description = "Numeric expression, cast to DOUBLE. NULL, 0 or a 
negative number returns NULL."
+    )
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLn {
+    signature: Signature,
+}
+
+unary_log_udf!(SparkLn, "ln", 0.0, fdlibm::log);
+
+/// Spark-compatible `log10` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#log10>
+#[user_doc(
+    doc_section(label = "Math Functions"),
+    description = "Returns the base 10 logarithm of `expr` as a DOUBLE. The 
argument is cast to \
+        DOUBLE. Unlike DataFusion's built-in `log10`, which returns -Infinity 
for 0 and NaN for a \
+        negative number, an input less than or equal to 0 returns NULL. 
Returns NULL if the input \
+        is NULL, and NaN if it is NaN.",
+    syntax_example = "log10(expr)",
+    sql_example = r#"```sql
+> SELECT log10(10);
+1
+> SELECT log10(-1);
+NULL
+```"#,
+    argument(
+        name = "expr",
+        description = "Numeric expression, cast to DOUBLE. NULL, 0 or a 
negative number returns NULL."
+    )
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLog10 {
+    signature: Signature,
+}
+
+unary_log_udf!(SparkLog10, "log10", 0.0, fdlibm::log10);
+
+/// Spark-compatible `log2` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#log2>
+#[user_doc(
+    doc_section(label = "Math Functions"),
+    description = "Returns the base 2 logarithm of `expr` as a DOUBLE. The 
argument is cast to \
+        DOUBLE. Like Spark, it is computed as `ln(expr) / ln(2)`, so it can 
differ from an exact \
+        base 2 logarithm in the last digit. Unlike DataFusion's built-in 
`log2`, which returns \
+        -Infinity for 0 and NaN for a negative number, an input less than or 
equal to 0 returns \
+        NULL. Returns NULL if the input is NULL, and NaN if it is NaN.",
+    syntax_example = "log2(expr)",
+    sql_example = r#"```sql
+> SELECT log2(2);
+1
+> SELECT log2(0);
+NULL
+```"#,
+    argument(
+        name = "expr",
+        description = "Numeric expression, cast to DOUBLE. NULL, 0 or a 
negative number returns NULL."
+    )
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLog2 {
+    signature: Signature,
+}
+
+// Spark computes `StrictMath.log(x) / StrictMath.log(2)` rather than a true
+// base 2 logarithm, which differs in the last bit for some inputs.
+unary_log_udf!(SparkLog2, "log2", 0.0, |x: f64| {
+    fdlibm::log(x) / fdlibm::log(2.0)
+});

Review Comment:
   `fdlibm::log(2.0)` is recomputed for every row. `StrictMath.log(2.0)` is 
exactly `LN_2`: the `2.0` row in `fdlibm::tests` expects `4604418534313441775`, 
which is `LN_2.to_bits()`. So the constant can replace the call, with `use 
std::f64::consts::LN_2;` at the top. An `assert_eq!(log(2.0), LN_2)` in 
`fdlibm::tests` would check that equivalence by name.
   
   ```suggestion
   // Spark computes `StrictMath.log(x) / StrictMath.log(2)` rather than a true
   // base 2 logarithm, which differs in the last bit for some inputs.
   // `StrictMath.log(2.0)` is exactly `LN_2`.
   unary_log_udf!(SparkLog2, "log2", 0.0, |x: f64| fdlibm::log(x) / LN_2);
   ```



##########
crates/sqllogictest/test_files/math/log1p.slt:
##########
@@ -8,7 +8,58 @@
 # For more information, please see:
 #   https://github.com/apache/datafusion/issues/15914
 
+
 ## Original Query: SELECT log1p(0);
 ## PySpark 3.5.5 Result: {'LOG1P(0)': 0.0, 'typeof(LOG1P(0))': 'double', 
'typeof(0)': 'int'}
-#query
-#SELECT log1p(0::int);
+query R
+SELECT log1p(0::int);
+----
+0
+
+query T
+SELECT arrow_typeof(log1p(0::int));
+----
+Float64
+
+query RR
+SELECT log1p(1::int), log1p(-0.5::double);
+----
+0.6931471805599453 -0.6931471805599453
+
+# Accurate near 0, where ln(1 + x) would round to 0
+query R
+SELECT log1p(1e-20::double);
+----
+0.00000000000000000001
+
+# Spark returns NULL, not -Infinity or NaN, for an input <= -1
+query RRR
+SELECT log1p(-1::int), log1p(-2::int), log1p('-Infinity'::double);
+----
+NULL NULL NULL
+
+query RRR
+SELECT log1p(NULL::double), log1p('NaN'::double), log1p('Infinity'::double);
+----
+NULL NaN Infinity
+
+query R
+SELECT log1p(a) FROM (VALUES (0::double), (-1::double), (NULL::double), 
(-0.5::double)) AS t(a);
+----
+0
+NULL
+NULL
+-0.6931471805599453
+
+# Spark uses StrictMath.log1p (fdlibm), which differs from Rust's f64::ln_1p in
+# the last digit for some inputs (f64::ln_1p gives 0.5130099025005953 here)
+query R
+SELECT log1p(0.670311110015402::double);
+----
+0.5130099025005954
+
+# -0.0 is above the asymptote, and log1p(-0.0) is -0.0
+query R
+SELECT log1p(-0.0::double);

Review Comment:
   In Spark, `-0.0` is a DECIMAL literal, and decimals have no negative zero. 
So `-0.0::double` is `0.0`, and Spark 4.2.0 returns `0.0` for this query, not 
`-0.0`. The PySpark runner flags this line. `'-0.0'::double` is `-0.0` in both 
engines. `ln.slt` line 43 has the same `ln(-0.0::double)`, which tests `0.0` in 
Spark.
   
   ```suggestion
   SELECT log1p('-0.0'::double);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to