andygrove opened a new issue, #198:
URL: https://github.com/apache/datafusion-spark/issues/198

   Part of #177 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `rpad(str, len[, pad])` returns `str` right-padded with `pad` to a length of 
`len`. If `str` is longer than `len`, it is truncated to its first `len` 
characters (or bytes). Available since Spark 1.5.0.
   
   `rpad` is registered via `RPadExpressionBuilder` (shared 
`PadExpressionBuilderBase` with `lpad`), which picks:
   
   - **String form** (`StringRPad`): `str` STRING, `len` INT, `pad` STRING 
(default `' '`). These are `ImplicitCastInputTypes`, so other `str`/`pad` types 
are cast to STRING and `len` to INT. Returns `str`'s type, collation included. 
Lengths count UTF-8 code points, so a surrogate pair is one character.
   - **Binary form** (`BinaryPad`, a `StaticInvoke` of `ByteArray.rpad`): used 
when `str` is BINARY and `pad` is absent or BINARY. The default pad is `X'00'`, 
lengths count bytes, and the result is BINARY. It is disabled when 
`spark.sql.legacy.lpadRpadAlwaysReturnString=true` (internal, default `false`, 
since 3.3.0).
   
   The function is null-intolerant: a NULL in any argument returns NULL.
   
   ### Examples
   
   ```sql
   SELECT rpad('hi', 5, '??');
   -- hi???
   SELECT rpad('hi', 1, '??');
   -- h
   SELECT rpad('hi', 5);
   -- 'hi   '
   SELECT rpad('hi', 5, 'xy');
   -- hixyx
   SELECT rpad('hi', -5, 'xy');
   -- (empty string)
   SELECT rpad('hi', 5, '');
   -- hi
   SELECT rpad('hello', 2);
   -- he
   SELECT hex(rpad(unhex('aabb'), 5));
   -- AABB000000
   SELECT hex(rpad(unhex('aabb'), 5, unhex('1122')));
   -- AABB112211
   SELECT hex(rpad(unhex('aa'), 6, unhex('1f2e')));
   -- AA1F2E1F2E1F
   SELECT hex(rpad(unhex('aabbcc'), 2, unhex('ff')));
   -- AABB
   SELECT hex(rpad(unhex(''), 6, unhex('')));
   -- (empty)
   SELECT rpad('abc', 5, x'57');   -- mixed types use the string form
   -- abcWW
   SELECT rpad(x'57', 5, 'abc');
   -- Wabca
   ```
   
   Unicode cases from `UTF8StringSuite` (calls to `UTF8String.rpad`):
   
   ```text
   rpad('数据砖头', 5, '????')   -> 数据砖头?
   rpad('数据砖头', 6, '孙行者') -> 数据砖头孙行
   rpad('数据砖头', 12, '孙行者') -> 数据砖头孙行者孙行者孙行
   rpad('数据砖头', 3, '')       -> 数据砖
   rpad('数据砖头', -10, '孙行者') -> ''
   ```
   
   ### Edge cases to test
   
   - NULL `str`, NULL `len`, or NULL `pad` returns NULL, for both forms.
   - Strings with `len <= 0` return `''`. When `len` is shorter than the input, 
the input is truncated to its first `len` characters.
   - An empty `pad` never pads. The result is the input, truncated to `len` 
characters if it is longer.
   - Partial repetition of a multi-character pad is counted in characters, not 
bytes (`'孙行者'` padding 2 characters gives `孙行`).
   - Supplementary-plane characters in `str` and in `pad` each count as one 
character. Combining marks count separately, so this is not grapheme-based.
   - Binary form:
     - With `len = 0` it returns empty bytes.
     - With `len < 0` Spark throws a raw `java.lang.NegativeArraySizeException` 
from `new byte[len]`, with or without an empty pad. This was derived from 
reading the code; Spark has no test for it, and the implementation has to 
decide whether to mirror it with an error.
     - With an empty pad it returns the first `min(len, bytes.length)` bytes.
     - The default pad is `0x00`.
   - Type dispatch: (binary, n) and (binary, n, binary) return BINARY. Every 
other combination returns STRING.
   - A non-numeric `len` such as `rpad('hi', 'invalid_length')` fails with 
`CAST_INVALID_INPUT` under ANSI mode (the default since 4.0). With ANSI off it 
returns NULL. This is handled by the analyzer's implicit cast.
   - The string form throws `ArithmeticException` (`Math.toIntExact`) if the 
result would be over 2^31-1 bytes.
   - Literal and column variants for each of `str`, `len` and `pad`.
   - Collations (4.0+): any collation is accepted, but padding ignores 
collation. The result type follows `str`. Two conflicting explicit collations 
is an analysis error.
   
   ### Differences across Spark versions
   
   - 3.4 / 3.5: as above (binary-aware since 3.3.0).
   - 4.0: collation-aware input types. ANSI is on by default, which affects the 
implicit cast of `len`. Semantics are unchanged for `UTF8_BINARY`.
   - 4.1 / 4.2: none found.
   
   ### Implementation notes
   
   - Spark source: 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala`
 (`RPadExpressionBuilder`, `PadExpressionBuilderBase`, `StringRPad`, 
`BinaryPad`). Kernels: `UTF8String.rpad` in 
`common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java` and 
`ByteArray.rpad` in 
`common/unsafe/src/main/java/org/apache/spark/unsafe/types/ByteArray.java`.
   - DataFusion core `rpad` (`datafusion/functions/src/unicode/rpad.rs`) counts 
code points as of apache/datafusion#21405 (DataFusion 54; it was grapheme-based 
before). Its string behaviour matches Spark for:
     - negative length (gives `''`);
     - truncation;
     - an empty fill.
   
     It differs in that:
     - it has no BINARY input;
     - `len` is Int64 and must be `<= i32::MAX`.
   
     The string path can delegate to core after coercing `len`. The binary path 
needs a new kernel, shared with `lpad`.
   - Module: `crates/spark/src/function/string/`, ideally in one `pad.rs` with 
`lpad`.
   - Tests: enable the commented-out queries in 
`crates/sqllogictest/test_files/string/rpad.slt` and add the binary, negative 
and Unicode cases above. Document with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to