andygrove opened a new issue, #200:
URL: https://github.com/apache/datafusion-spark/issues/200
Part of #177 and apache/datafusion-comet#7300.
### What the function does
`find_in_set(str, str_array)` returns the 1-based index of `str` in the
comma-delimited list `str_array`. It returns 0 if `str` is not found or if
`str` contains a comma. Available since Spark 1.5.0.
Both arguments are STRING (`StringTypeWithCollation(supportsTrimCollation =
true)`) via `ImplicitCastInputTypes`, so non-strings are cast to STRING. The
return type is INT. It is null-intolerant: NULL in either argument returns
NULL. Only ASCII `,` (0x2C) is a delimiter. Comparison is exact byte equality
under `UTF8_BINARY`, with no trimming of the list elements.
### Examples
```sql
SELECT find_in_set('ab', 'abc,b,ab,c,def');
-- 3
SELECT find_in_set('abf', 'abc,b,ab,c,def');
-- 0
SELECT find_in_set('ab,', 'abc,b,ab,c,def'); -- word contains a comma
-- 0
SELECT find_in_set('ab', 'ab');
-- 1
SELECT find_in_set('b', 'a,b');
-- 2
SELECT find_in_set('ab', 'ab,abc,b,ab,c,def'); -- first match wins
-- 1
SELECT find_in_set('ab', ',,,ab,abc,b,ab,c,def'); -- empty elements count
-- 4
SELECT find_in_set('', ',ab,abc,b,ab,c,def'); -- empty word matches an
empty element
-- 1
SELECT find_in_set('ab', '数据砖头,abc,b,ab,c,def');
-- 4
SELECT find_in_set('ab', NULL);
-- NULL
```
### Edge cases to test
- NULL `str`, NULL `str_array`, or both returns NULL.
- Empty `str`: it matches the first empty element. That includes a leading,
trailing or doubled comma, and an empty `str_array` (`find_in_set('', '')`
returns 1; derived from `UTF8String.findInSet`, whose final check compares the
empty last segment). If no element is empty, the result is 0.
- `str` containing a comma always returns 0, even if the list contains the
same text.
- Elements with surrounding spaces do not match (`find_in_set('b', 'a, b')`
returns 0).
- Multi-byte UTF-8 elements, and the returned index counting elements rather
than bytes.
- Case sensitivity under `UTF8_BINARY` (`find_in_set('AB', 'ab')` returns 0).
- Literal and column variants of both arguments.
- Collations (4.0+): under `UTF8_LCASE` and ICU collations each element is
compared with `semanticEquals` for that collation, so case-insensitive matches
count. `*_RTRIM` collations ignore trailing spaces. Supporting `UTF8_BINARY`
only and rejecting other collations is acceptable for a first version.
### Differences across Spark versions
- 3.4 / 3.5: as above.
- 4.0: collation-aware path via `CollationSupport.FindInSet`; `UTF8_BINARY`
is unchanged.
- 4.1 / 4.2: none found.
### Implementation notes
- Spark source:
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala`
(`FindInSet`). Kernel: `UTF8String.findInSet` in
`common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java`.
Collation path: `CollationAwareUTF8String.findInSet`.
- DataFusion core has `find_in_set`
(`datafusion/functions/src/unicode/find_in_set.rs`). It uses
`str_list.split(',').position(|s| s == string)`, which gives the same result
for `UTF8_BINARY` input, including the comma-in-word, empty-word and empty-list
cases. The differences are:
- it returns Int64 for `LargeUtf8` input, while Spark always returns INT;
- there is no collation support.
A thin wrapper that fixes the return type to Int32 (and coerces non-string
arguments to STRING) is probably enough. Check the scalar/array fast paths for
the same semantics.
- Module: `crates/spark/src/function/string/find_in_set.rs`.
- Tests: `crates/sqllogictest/test_files/string/find_in_set.slt` exists as a
placeholder; fill it with the cases above. Document with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]