andygrove opened a new issue, #207:
URL: https://github.com/apache/datafusion-spark/issues/207

   Part of #177 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `lower(str)` returns `str` with every character converted to lowercase. 
`lcase` is a registry alias
   (`expression[Lower]("lcase", true)`), and `prettyName` shows whichever alias 
the query used. Available since
   Spark 1.0.1.
   
   The input type is `STRING` (any collation from 4.0 on). `Lower` mixes in 
`ImplicitCastInputTypes`, so the analyzer
   casts other atomic inputs (numbers, binary, dates, ...) to `STRING` before 
the function runs. The return type is the
   input's string type (collation kept), the function is null-intolerant, and 
it is deterministic.
   
   The mapping is a full Unicode lowercase mapping on code points, and it is 
context-sensitive for Greek capital sigma:
   `Σ` becomes `ς` at the end of a word and `σ` everywhere else. One code point 
can map to several (`İ` -> `i` + U+0307).
   
   ### Examples
   
   ```sql
   SELECT lower('SparkSql');
   -- sparksql
   SELECT lcase('SparkSql');
   -- sparksql
   SELECT lower('ÄBĆΔE');
   -- äbćδe
   SELECT lower('ΣΑΛΑΤΑ');        -- leading sigma
   -- σαλατα
   SELECT lower('ΘΑΛΑΣΣΙΝΟΣ');    -- word-final sigma
   -- θαλασσινος
   SELECT lower('Σ');             -- lone sigma is not word-final
   -- σ
   SELECT lower('İ');             -- U+0130
   -- i̇                     (i + U+0307 combining dot, 3 bytes)
   SELECT lower('FIDELİO');
   -- fideli̇o
   SELECT lower('ß fi ffi ff st ῗ');   -- no lowercase mapping, unchanged
   -- ß fi ffi ff st ῗ
   SELECT lower('𐐅');             -- surrogate pair (Deseret capital)
   -- 𐐭
   SELECT lower('a🙃B🙃c');
   -- a🙃b🙃c
   SELECT lower(NULL);
   -- NULL
   ```
   
   (The non-trivial outputs come from `CollationSupportSuite.testLower`. The 
suite checks both the ICU and the JVM code
   paths and expects the same result from each.)
   
   ### Edge cases to test
   
   - NULL input returns NULL. Empty string returns empty string. Test scalar 
and array input, and `Utf8`, `LargeUtf8`
     and `Utf8View`.
   - Final sigma context, which depends on the characters before and after 
(case-ignorable characters are skipped):
     `ΘΑΛΑΣΣΙΝΟΣ`, `Σ` alone, and `ΣΑΣ` followed by a space or punctuation. 
Mapping each character on its own
     gets these wrong.
   - Lowercase sigmas already in the input are kept: `ςαλατα` stays `ςαλατα` 
and `θαλασσινοσ` stays `θαλασσινοσ`.
   - One-to-many mappings that make the output longer: `İ` -> `i̇`, which grows 
from 2 bytes to 3.
   - Turkish: `I` -> `i`, never dotless `ı`. Spark uses root-locale rules, 
apart from the JVM default-locale caveat below.
   - Supplementary-plane characters (surrogate pairs): `𐐅` -> `𐐭`. `𝔸` and 
emoji are unchanged.
   - All-ASCII input versus mixed input. Spark takes a per-byte fast path only 
when the whole string is ASCII.
   - Numeric and binary inputs reach the function only after an analyzer cast 
to string. Spark's slow path decodes
     invalid UTF-8 bytes (possible after `CAST(binary AS STRING)`) as U+FFFD.
   - Collations (4.0+): `UTF8_BINARY`, `UTF8_LCASE`, `UNICODE`, `UNICODE_CI` 
and locale collations. For ICU locale
     collations, non-ASCII input uses that locale, but all-ASCII input always 
takes the locale-independent fast path. A
     first version can support only `UTF8_BINARY`.
   - `spark.sql.ansi.enabled` has no effect.
   
   ### Differences across Spark versions
   
   - 3.4 / 3.5: `UTF8String.toLowerCase()`. All-ASCII input uses a per-byte 
fast path. Everything else falls back to
     `java.lang.String.toLowerCase()`, which uses the JVM default locale. There 
are no collations.
   - 4.0+: dispatches through `CollationSupport.Lower.exec(v, collationId, 
useICU)`. `spark.sql.icu.caseMappings.enabled`
     (default `true`, since 4.0.0) picks ICU `UCharacter.toLowerCase` instead 
of the JVM for `UTF8_BINARY`. `UTF8_LCASE`
     always uses ICU. Other collations use ICU with the collation's locale.
   - 4.1 / 4.2: no behavior change found.
   
   ### Implementation notes
   
   - Spark: 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala`
 (`Lower`),
     
`common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java`
 (`Lower`),
     
`common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java`
 (`toLowerCase`), and
     
`common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java` 
(`toLowerCase`, `toLowerCaseAscii`).
   - DataFusion core has `lower` (`datafusion/functions/src/string/lower.rs`, 
`string/common.rs::to_lower`). It uses
     Rust's `str::to_lowercase`, which is locale-insensitive, applies the full 
mapping (`İ` -> `i̇`) and
     implements the Unicode `Final_Sigma` rule. That is the same model as 
Spark's default for `UTF8_BINARY`. The
     remaining differences are:
     - Unicode version: Rust std vs ICU4J vs the JDK.
     - Spark's dependence on the JVM default locale (`tr`, `az` or `lt` change 
non-ASCII results).
     - Collation locales.
   
     Wrapping DataFusion's kernel should work for `UTF8_BINARY`. Most of the 
work is pinning the cases above in tests.
   - Module: `crates/spark/src/function/string/` (`lower.rs`), registered as 
`lower`. Expose `lcase` through
     `aliases()`.
   - Tests: add `crates/sqllogictest/test_files/string/lower.slt` (follow 
`elt.slt`). Document the function with
     `#[user_doc]` and run `dev/update_function_docs.sh`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to