mbutrovich commented on code in PR #328:
URL: https://github.com/apache/datafusion-spark/pull/328#discussion_r4605623108
##########
crates/spark/src/function/string/contains.rs:
##########
@@ -0,0 +1,651 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF)
under one or more contributor license agreements
+
+use std::sync::Arc;
+
+use arrow::array::{Array, ArrayRef, BooleanArray, Scalar};
+use arrow::compute::kernels::cast::cast;
+use arrow::compute::kernels::comparison::contains as arrow_contains;
+use arrow::datatypes::DataType;
+use datafusion_common::utils::take_function_args;
+use datafusion_common::{Result, ScalarValue, exec_err, plan_err};
+use datafusion_expr::{
+ ColumnarValue, Documentation, ScalarFunctionArgs, ScalarUDFImpl, Signature,
+ Volatility,
+};
+use datafusion_macros::user_doc;
+
+/// Spark-compatible `contains` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#contains>
+#[user_doc(
+ doc_section(label = "String Functions"),
+ description = "Returns true if `right` is found inside `left`.
DataFusion's own `contains` takes only \
+ strings; this one also accepts binary. When both arguments are binary
the match is on bytes. \
+ When one argument is a string and the other is binary, both are
compared as bytes, which is \
+ the same as Spark casting the binary to a string. Integer, boolean and
date arguments are \
+ cast to a string first. The match is case-sensitive, and an empty
`right` is found in every \
+ value, including an empty `left`. Returns NULL if either input is
NULL.",
+ syntax_example = "contains(left, right)",
+ sql_example = r#"```sql
+> SELECT contains('Spark SQL', 'Spark');
+true
+> SELECT contains('Spark SQL', 'SPARK');
+false
+> SELECT contains(x'537061726b2053514c', x'537061726b');
+true
+```"#,
+ argument(
+ name = "left",
+ description = "String or binary expression to search in. NULL returns
NULL."
+ ),
+ argument(
+ name = "right",
+ description = "String or binary expression to search for. NULL returns
NULL."
+ )
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkContains {
+ signature: Signature,
+}
+
+impl Default for SparkContains {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkContains {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::user_defined(Volatility::Immutable),
+ }
+ }
+}
+
+/// How an argument type takes part in `contains`.
+#[derive(Clone, Copy, PartialEq, Eq)]
+enum Kind {
+ Null,
+ String,
+ Binary,
+}
+
+/// Classifies an argument type, returning the type it is coerced to before
+/// unification. Types Spark implicitly casts to a string and whose Arrow
string
+/// form matches Spark's become `Utf8`.
+fn classify(name: &str, dt: &DataType) -> Result<(Kind, DataType)> {
+ match dt {
+ DataType::Null => Ok((Kind::Null, DataType::Null)),
+ DataType::Utf8 | DataType::LargeUtf8 | DataType::Utf8View => {
+ Ok((Kind::String, dt.clone()))
+ }
+ DataType::Binary | DataType::LargeBinary | DataType::BinaryView => {
+ Ok((Kind::Binary, dt.clone()))
+ }
+ DataType::FixedSizeBinary(_) => Ok((Kind::Binary, DataType::Binary)),
+ DataType::Dictionary(_, value) => classify(name, value),
+ DataType::Int8
+ | DataType::Int16
+ | DataType::Int32
+ | DataType::Int64
+ | DataType::UInt8
+ | DataType::UInt16
+ | DataType::UInt32
+ | DataType::UInt64
+ | DataType::Boolean
+ | DataType::Date32 => Ok((Kind::String, DataType::Utf8)),
+ other => plan_err!(
+ "Unsupported Data Type: Spark `{name}` function expects STRING or
BINARY arguments, got {other}"
+ ),
Review Comment:
> `contains.slt` checks that a DOUBLE is rejected and records Spark 4.2.0's
results as commented-out queries, including the decimal case that depends on
ANSI mode.
Two things would stop these records from passing once #347 is fixed and
someone uncomments them ([`contains.slt`
L133-L144](https://github.com/apache/datafusion-spark/blob/dd706b37fc17e816b04afb1c43090ab9156a4d54/crates/sqllogictest/test_files/string/contains.slt?plain=1#L133-L144)).
- The decimal record expects `false`, which is Spark's result with ANSI mode
on. The file never turns ANSI mode on, so this record would run with it off,
where Spark returns `true`.
- `1.5D` is Spark syntax, from my earlier comment. DataFusion fails to parse
it (`ParserError("Expected: ), found: D")`). `1.5::double` gives `true` in
Spark 4.2.0 in both modes, as the `statement error` query above it already uses.
Could we record the decimal case in both modes and use DataFusion syntax?
For example:
```
# Spark 4.2.0 returns true for these
# query BB
# SELECT contains(1.5::double, '.'), contains(TIMESTAMP '2024-01-01
00:00:00', ' ');
# ----
# true true
# Spark 4.2.0 formats a DECIMAL as a string differently in the two ANSI modes
# query B
# SELECT contains(CAST(0.0000001 AS DECIMAL(10,8)), 'E');
# ----
# true
#
# statement ok
# set datafusion.execution.enable_ansi_mode = true;
#
# query B
# SELECT contains(CAST(0.0000001 AS DECIMAL(10,8)), 'E');
# ----
# false
#
# statement ok
# set datafusion.execution.enable_ansi_mode = false;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]