andygrove opened a new issue, #200:
URL: https://github.com/apache/datafusion-spark/issues/200

   Part of #177 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `find_in_set(str, str_array)` returns the 1-based index of `str` in the 
comma-delimited list `str_array`. It returns 0 if `str` is not found or if 
`str` contains a comma. Available since Spark 1.5.0.
   
   Both arguments are STRING (`StringTypeWithCollation(supportsTrimCollation = 
true)`) via `ImplicitCastInputTypes`, so non-strings are cast to STRING. The 
return type is INT. It is null-intolerant: NULL in either argument returns 
NULL. Only ASCII `,` (0x2C) is a delimiter. Comparison is exact byte equality 
under `UTF8_BINARY`, with no trimming of the list elements.
   
   ### Examples
   
   ```sql
   SELECT find_in_set('ab', 'abc,b,ab,c,def');
   -- 3
   SELECT find_in_set('abf', 'abc,b,ab,c,def');
   -- 0
   SELECT find_in_set('ab,', 'abc,b,ab,c,def');   -- word contains a comma
   -- 0
   SELECT find_in_set('ab', 'ab');
   -- 1
   SELECT find_in_set('b', 'a,b');
   -- 2
   SELECT find_in_set('ab', 'ab,abc,b,ab,c,def');   -- first match wins
   -- 1
   SELECT find_in_set('ab', ',,,ab,abc,b,ab,c,def'); -- empty elements count
   -- 4
   SELECT find_in_set('', ',ab,abc,b,ab,c,def');     -- empty word matches an 
empty element
   -- 1
   SELECT find_in_set('ab', '数据砖头,abc,b,ab,c,def');
   -- 4
   SELECT find_in_set('ab', NULL);
   -- NULL
   ```
   
   ### Edge cases to test
   
   - NULL `str`, NULL `str_array`, or both returns NULL.
   - Empty `str`: it matches the first empty element. That includes a leading, 
trailing or doubled comma, and an empty `str_array` (`find_in_set('', '')` 
returns 1; derived from `UTF8String.findInSet`, whose final check compares the 
empty last segment). If no element is empty, the result is 0.
   - `str` containing a comma always returns 0, even if the list contains the 
same text.
   - Elements with surrounding spaces do not match (`find_in_set('b', 'a, b')` 
returns 0).
   - Multi-byte UTF-8 elements, and the returned index counting elements rather 
than bytes.
   - Case sensitivity under `UTF8_BINARY` (`find_in_set('AB', 'ab')` returns 0).
   - Literal and column variants of both arguments.
   - Collations (4.0+): under `UTF8_LCASE` and ICU collations each element is 
compared with `semanticEquals` for that collation, so case-insensitive matches 
count. `*_RTRIM` collations ignore trailing spaces. Supporting `UTF8_BINARY` 
only and rejecting other collations is acceptable for a first version.
   
   ### Differences across Spark versions
   
   - 3.4 / 3.5: as above.
   - 4.0: collation-aware path via `CollationSupport.FindInSet`; `UTF8_BINARY` 
is unchanged.
   - 4.1 / 4.2: none found.
   
   ### Implementation notes
   
   - Spark source: 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala`
 (`FindInSet`). Kernel: `UTF8String.findInSet` in 
`common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java`. 
Collation path: `CollationAwareUTF8String.findInSet`.
   - DataFusion core has `find_in_set` 
(`datafusion/functions/src/unicode/find_in_set.rs`). It uses 
`str_list.split(',').position(|s| s == string)`, which gives the same result 
for `UTF8_BINARY` input, including the comma-in-word, empty-word and empty-list 
cases. The differences are:
     - it returns Int64 for `LargeUtf8` input, while Spark always returns INT;
     - there is no collation support.
   
     A thin wrapper that fixes the return type to Int32 (and coerces non-string 
arguments to STRING) is probably enough. Check the scalar/array fast paths for 
the same semantics.
   - Module: `crates/spark/src/function/string/find_in_set.rs`.
   - Tests: `crates/sqllogictest/test_files/string/find_in_set.slt` exists as a 
placeholder; fill it with the cases above. Document with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to