getChan opened a new issue, #24788:
URL: https://github.com/apache/datafusion/issues/24788
### Is your feature request related to a problem or challenge?
`character_length` currently accepts `Binary`, `LargeBinary`, and
`BinaryView` through an implicit cast to a string type rather than handling
them with binary-string semantics. Its aliases `char_length` and `length`
inherit the same behavior.
This produces incorrect results for valid multi-byte UTF-8 and rejects
arbitrary binary data:
```sql
SELECT
character_length(X'C3A9'),
char_length(X'C3A9'),
length(X'C3A9');
```
All three currently return `1` because `X'C3A9'` is cast to UTF-8 and
interpreted as the single character `é`. The type-coercion plan contains:
```text
character_length(CAST(Binary("195,169") AS Utf8View))
```
Likewise, arbitrary binary data is rejected:
```sql
SELECT character_length(X'FF');
```
```text
Encountered non UTF-8 data: invalid utf-8 sequence
```
SQL:1999 defines `CHAR_LENGTH` / `CHARACTER_LENGTH` over a BLOB as its
length in octets. Under those semantics, the expected results are:
```sql
character_length(X'C3A9') -- 2
character_length(X'FF') -- 1
```
ASCII binary values happen to return the expected value today because their
UTF-8 character count equals their byte count.
### Describe the solution you'd like
Add a native binary path to `character_length` for `Binary`, `LargeBinary`,
and `BinaryView`:
- Return the number of bytes without UTF-8 validation or decoding.
- Preserve the existing character-count semantics for `Utf8`, `LargeUtf8`,
and `Utf8View`.
- Preserve the existing return-width convention: `Binary` / `BinaryView`
return `Int32`, and `LargeBinary` returns `Int64`.
- Ensure the exact binary signature is selected before the existing implicit
`Any -> String` coercion.
- Apply the same behavior to the `char_length` and `length` aliases
automatically.
This changes the result for binary values containing valid multi-byte UTF-8,
but aligns binary inputs with SQL BLOB semantics.
### Describe alternatives you've considered
Explicitly casting Binary to `VARCHAR` does not provide binary semantics: it
still counts Unicode characters and fails for invalid UTF-8.
A byte-count workaround based on hexadecimal encoding materializes twice as
many bytes merely to calculate the length:
```sql
length(encode(binary_col, 'hex')) / 2
```
### Additional context
Related to #24765, which adds native binary support to `octet_length` and
notes the corresponding `bit_length` gap. `character_length` differs because it
currently accepts many binary values but silently applies character-string
semantics through implicit coercion.
- SQL:1999 BLOB length functions:
https://sql-99.readthedocs.io/en/latest/chapters/05.html
- PostgreSQL binary string functions:
https://www.postgresql.org/docs/current/functions-binarystring.html
- Existing DataFusion coverage documenting the UTF-8 coercion:
`datafusion/sqllogictest/test_files/binary.slt`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
