andygrove opened a new issue, #207:
URL: https://github.com/apache/datafusion-spark/issues/207
Part of #177 and apache/datafusion-comet#7300.
### What the function does
`lower(str)` returns `str` with every character converted to lowercase.
`lcase` is a registry alias
(`expression[Lower]("lcase", true)`), and `prettyName` shows whichever alias
the query used. Available since
Spark 1.0.1.
The input type is `STRING` (any collation from 4.0 on). `Lower` mixes in
`ImplicitCastInputTypes`, so the analyzer
casts other atomic inputs (numbers, binary, dates, ...) to `STRING` before
the function runs. The return type is the
input's string type (collation kept), the function is null-intolerant, and
it is deterministic.
The mapping is a full Unicode lowercase mapping on code points, and it is
context-sensitive for Greek capital sigma:
`Σ` becomes `ς` at the end of a word and `σ` everywhere else. One code point
can map to several (`İ` -> `i` + U+0307).
### Examples
```sql
SELECT lower('SparkSql');
-- sparksql
SELECT lcase('SparkSql');
-- sparksql
SELECT lower('ÄBĆΔE');
-- äbćδe
SELECT lower('ΣΑΛΑΤΑ'); -- leading sigma
-- σαλατα
SELECT lower('ΘΑΛΑΣΣΙΝΟΣ'); -- word-final sigma
-- θαλασσινος
SELECT lower('Σ'); -- lone sigma is not word-final
-- σ
SELECT lower('İ'); -- U+0130
-- i̇ (i + U+0307 combining dot, 3 bytes)
SELECT lower('FIDELİO');
-- fideli̇o
SELECT lower('ß fi ffi ff st ῗ'); -- no lowercase mapping, unchanged
-- ß fi ffi ff st ῗ
SELECT lower('𐐅'); -- surrogate pair (Deseret capital)
-- 𐐭
SELECT lower('a🙃B🙃c');
-- a🙃b🙃c
SELECT lower(NULL);
-- NULL
```
(The non-trivial outputs come from `CollationSupportSuite.testLower`. The
suite checks both the ICU and the JVM code
paths and expects the same result from each.)
### Edge cases to test
- NULL input returns NULL. Empty string returns empty string. Test scalar
and array input, and `Utf8`, `LargeUtf8`
and `Utf8View`.
- Final sigma context, which depends on the characters before and after
(case-ignorable characters are skipped):
`ΘΑΛΑΣΣΙΝΟΣ`, `Σ` alone, and `ΣΑΣ` followed by a space or punctuation.
Mapping each character on its own
gets these wrong.
- Lowercase sigmas already in the input are kept: `ςαλατα` stays `ςαλατα`
and `θαλασσινοσ` stays `θαλασσινοσ`.
- One-to-many mappings that make the output longer: `İ` -> `i̇`, which grows
from 2 bytes to 3.
- Turkish: `I` -> `i`, never dotless `ı`. Spark uses root-locale rules,
apart from the JVM default-locale caveat below.
- Supplementary-plane characters (surrogate pairs): `𐐅` -> `𐐭`. `𝔸` and
emoji are unchanged.
- All-ASCII input versus mixed input. Spark takes a per-byte fast path only
when the whole string is ASCII.
- Numeric and binary inputs reach the function only after an analyzer cast
to string. Spark's slow path decodes
invalid UTF-8 bytes (possible after `CAST(binary AS STRING)`) as U+FFFD.
- Collations (4.0+): `UTF8_BINARY`, `UTF8_LCASE`, `UNICODE`, `UNICODE_CI`
and locale collations. For ICU locale
collations, non-ASCII input uses that locale, but all-ASCII input always
takes the locale-independent fast path. A
first version can support only `UTF8_BINARY`.
- `spark.sql.ansi.enabled` has no effect.
### Differences across Spark versions
- 3.4 / 3.5: `UTF8String.toLowerCase()`. All-ASCII input uses a per-byte
fast path. Everything else falls back to
`java.lang.String.toLowerCase()`, which uses the JVM default locale. There
are no collations.
- 4.0+: dispatches through `CollationSupport.Lower.exec(v, collationId,
useICU)`. `spark.sql.icu.caseMappings.enabled`
(default `true`, since 4.0.0) picks ICU `UCharacter.toLowerCase` instead
of the JVM for `UTF8_BINARY`. `UTF8_LCASE`
always uses ICU. Other collations use ICU with the collation's locale.
- 4.1 / 4.2: no behavior change found.
### Implementation notes
- Spark:
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala`
(`Lower`),
`common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java`
(`Lower`),
`common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java`
(`toLowerCase`), and
`common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java`
(`toLowerCase`, `toLowerCaseAscii`).
- DataFusion core has `lower` (`datafusion/functions/src/string/lower.rs`,
`string/common.rs::to_lower`). It uses
Rust's `str::to_lowercase`, which is locale-insensitive, applies the full
mapping (`İ` -> `i̇`) and
implements the Unicode `Final_Sigma` rule. That is the same model as
Spark's default for `UTF8_BINARY`. The
remaining differences are:
- Unicode version: Rust std vs ICU4J vs the JDK.
- Spark's dependence on the JVM default locale (`tr`, `az` or `lt` change
non-ASCII results).
- Collation locales.
Wrapping DataFusion's kernel should work for `UTF8_BINARY`. Most of the
work is pinning the cases above in tests.
- Module: `crates/spark/src/function/string/` (`lower.rs`), registered as
`lower`. Expose `lcase` through
`aliases()`.
- Tests: add `crates/sqllogictest/test_files/string/lower.slt` (follow
`elt.slt`). Document the function with
`#[user_doc]` and run `dev/update_function_docs.sh`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]