andygrove opened a new issue, #198:
URL: https://github.com/apache/datafusion-spark/issues/198
Part of #177 and apache/datafusion-comet#7300.
### What the function does
`rpad(str, len[, pad])` returns `str` right-padded with `pad` to a length of
`len`. If `str` is longer than `len`, it is truncated to its first `len`
characters (or bytes). Available since Spark 1.5.0.
`rpad` is registered via `RPadExpressionBuilder` (shared
`PadExpressionBuilderBase` with `lpad`), which picks:
- **String form** (`StringRPad`): `str` STRING, `len` INT, `pad` STRING
(default `' '`). These are `ImplicitCastInputTypes`, so other `str`/`pad` types
are cast to STRING and `len` to INT. Returns `str`'s type, collation included.
Lengths count UTF-8 code points, so a surrogate pair is one character.
- **Binary form** (`BinaryPad`, a `StaticInvoke` of `ByteArray.rpad`): used
when `str` is BINARY and `pad` is absent or BINARY. The default pad is `X'00'`,
lengths count bytes, and the result is BINARY. It is disabled when
`spark.sql.legacy.lpadRpadAlwaysReturnString=true` (internal, default `false`,
since 3.3.0).
The function is null-intolerant: a NULL in any argument returns NULL.
### Examples
```sql
SELECT rpad('hi', 5, '??');
-- hi???
SELECT rpad('hi', 1, '??');
-- h
SELECT rpad('hi', 5);
-- 'hi '
SELECT rpad('hi', 5, 'xy');
-- hixyx
SELECT rpad('hi', -5, 'xy');
-- (empty string)
SELECT rpad('hi', 5, '');
-- hi
SELECT rpad('hello', 2);
-- he
SELECT hex(rpad(unhex('aabb'), 5));
-- AABB000000
SELECT hex(rpad(unhex('aabb'), 5, unhex('1122')));
-- AABB112211
SELECT hex(rpad(unhex('aa'), 6, unhex('1f2e')));
-- AA1F2E1F2E1F
SELECT hex(rpad(unhex('aabbcc'), 2, unhex('ff')));
-- AABB
SELECT hex(rpad(unhex(''), 6, unhex('')));
-- (empty)
SELECT rpad('abc', 5, x'57'); -- mixed types use the string form
-- abcWW
SELECT rpad(x'57', 5, 'abc');
-- Wabca
```
Unicode cases from `UTF8StringSuite` (calls to `UTF8String.rpad`):
```text
rpad('数据砖头', 5, '????') -> 数据砖头?
rpad('数据砖头', 6, '孙行者') -> 数据砖头孙行
rpad('数据砖头', 12, '孙行者') -> 数据砖头孙行者孙行者孙行
rpad('数据砖头', 3, '') -> 数据砖
rpad('数据砖头', -10, '孙行者') -> ''
```
### Edge cases to test
- NULL `str`, NULL `len`, or NULL `pad` returns NULL, for both forms.
- Strings with `len <= 0` return `''`. When `len` is shorter than the input,
the input is truncated to its first `len` characters.
- An empty `pad` never pads. The result is the input, truncated to `len`
characters if it is longer.
- Partial repetition of a multi-character pad is counted in characters, not
bytes (`'孙行者'` padding 2 characters gives `孙行`).
- Supplementary-plane characters in `str` and in `pad` each count as one
character. Combining marks count separately, so this is not grapheme-based.
- Binary form:
- With `len = 0` it returns empty bytes.
- With `len < 0` Spark throws a raw `java.lang.NegativeArraySizeException`
from `new byte[len]`, with or without an empty pad. This was derived from
reading the code; Spark has no test for it, and the implementation has to
decide whether to mirror it with an error.
- With an empty pad it returns the first `min(len, bytes.length)` bytes.
- The default pad is `0x00`.
- Type dispatch: (binary, n) and (binary, n, binary) return BINARY. Every
other combination returns STRING.
- A non-numeric `len` such as `rpad('hi', 'invalid_length')` fails with
`CAST_INVALID_INPUT` under ANSI mode (the default since 4.0). With ANSI off it
returns NULL. This is handled by the analyzer's implicit cast.
- The string form throws `ArithmeticException` (`Math.toIntExact`) if the
result would be over 2^31-1 bytes.
- Literal and column variants for each of `str`, `len` and `pad`.
- Collations (4.0+): any collation is accepted, but padding ignores
collation. The result type follows `str`. Two conflicting explicit collations
is an analysis error.
### Differences across Spark versions
- 3.4 / 3.5: as above (binary-aware since 3.3.0).
- 4.0: collation-aware input types. ANSI is on by default, which affects the
implicit cast of `len`. Semantics are unchanged for `UTF8_BINARY`.
- 4.1 / 4.2: none found.
### Implementation notes
- Spark source:
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala`
(`RPadExpressionBuilder`, `PadExpressionBuilderBase`, `StringRPad`,
`BinaryPad`). Kernels: `UTF8String.rpad` in
`common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java` and
`ByteArray.rpad` in
`common/unsafe/src/main/java/org/apache/spark/unsafe/types/ByteArray.java`.
- DataFusion core `rpad` (`datafusion/functions/src/unicode/rpad.rs`) counts
code points as of apache/datafusion#21405 (DataFusion 54; it was grapheme-based
before). Its string behaviour matches Spark for:
- negative length (gives `''`);
- truncation;
- an empty fill.
It differs in that:
- it has no BINARY input;
- `len` is Int64 and must be `<= i32::MAX`.
The string path can delegate to core after coercing `len`. The binary path
needs a new kernel, shared with `lpad`.
- Module: `crates/spark/src/function/string/`, ideally in one `pad.rs` with
`lpad`.
- Tests: enable the commented-out queries in
`crates/sqllogictest/test_files/string/rpad.slt` and add the binary, negative
and Unicode cases above. Document with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]