andygrove opened a new issue, #197:
URL: https://github.com/apache/datafusion-spark/issues/197

   Part of #177 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `lpad(str, len[, pad])` returns `str` left-padded with `pad` to a length of 
`len`. If `str` is longer than `len`, it is truncated to its first `len` 
characters (or bytes). Available since Spark 1.5.0.
   
   `lpad` is registered through `LPadExpressionBuilder`, which picks one of two 
implementations at analysis time:
   
   - **String form** (`StringLPad`): `str` STRING, `len` INT, `pad` STRING 
(default `' '`). These are `ImplicitCastInputTypes`, so numeric or other 
`str`/`pad` values are cast to STRING and `len` is cast to INT. The return type 
is `str`'s type, collation included. Length is counted in characters, where a 
character is one UTF-8 code point, so a surrogate pair (4-byte UTF-8) counts as 
one character.
   - **Binary form** (`BinaryPad`, a `StaticInvoke` of `ByteArray.lpad`): used 
when `str` is BINARY and either there is no `pad` or `pad` is also BINARY. The 
default `pad` is `X'00'`. Length is counted in bytes and the result is BINARY. 
This form is used only when `spark.sql.legacy.lpadRpadAlwaysReturnString` is 
`false` (internal, default `false`, added in 3.3.0). When it is `true`, binary 
input goes through the string form.
   
   The function is null-intolerant: a NULL in any argument returns NULL.
   
   ### Examples
   
   ```sql
   SELECT lpad('hi', 5, '??');
   -- ???hi
   SELECT lpad('hi', 1, '??');
   -- h
   SELECT lpad('hi', 5);
   --    hi
   SELECT lpad('hi', 5, 'xy');
   -- xyxhi
   SELECT lpad('hi', -5, 'xy');
   -- (empty string)
   SELECT lpad('hi', 5, '');
   -- hi
   SELECT lpad('hello', 2);
   -- he
   SELECT hex(lpad(unhex('aabb'), 5));
   -- 000000AABB
   SELECT hex(lpad(unhex('aabb'), 5, unhex('1122')));
   -- 112211AABB
   SELECT hex(lpad(unhex('aabbcc'), 2));
   -- AABB
   SELECT hex(lpad(unhex('aa'), 6, unhex('1f2e')));
   -- 1F2E1F2E1FAA
   SELECT hex(lpad(unhex('aabbcc'), 6, unhex('')));
   -- AABBCC
   SELECT lpad('abc', 5, x'57');   -- mixed types use the string form
   -- WWabc
   SELECT lpad(x'57', 5, 'abc');
   -- abcaW
   ```
   
   Unicode cases from `UTF8StringSuite` (calls to `UTF8String.lpad`):
   
   ```text
   lpad('数据砖头', 3, '????')   -> 数据砖
   lpad('数据砖头', 6, '孙行者') -> 孙行数据砖头
   lpad('数据砖头', 12, '孙行者') -> 孙行者孙行者孙行数据砖头
   lpad('数据砖头', -10, '')     -> ''
   lpad('', 7, '?????')          -> ???????
   ```
   
   ### Edge cases to test
   
   - NULL `str`, NULL `len`, or NULL `pad` returns NULL, for both the string 
and the binary form.
   - Strings with `len <= 0` return `''`. When `len` is less than the input 
length, the input is truncated to its first `len` characters (truncation keeps 
the left end, even though this is *left* padding).
   - An empty `pad` never pads. The result is `substring(str, 0, len)`: the 
input unchanged if it is shorter than `len`, otherwise truncated.
   - A multi-character `pad` is repeated and the last copy is cut short, 
counting characters (`'xy'` padding to 5 gives `xyxhi`).
   - Multi-byte and supplementary characters: `len` and the pad slicing count 
code points, not bytes or graphemes. Combining sequences (e.g. `e` + U+0301) 
count as 2.
   - Binary form:
     - With `len = 0` it returns empty bytes.
     - With `len < 0` Spark's `ByteArray.lpad` does `new byte[len]` and throws 
a raw `java.lang.NegativeArraySizeException`. This applies with an empty pad 
too (`padWithEmptyPattern` uses `min(bytes.length, len)`). There is no error 
condition for this; the implementation needs to decide whether to mirror it 
with an execution error. This was derived from reading the code; Spark has no 
test for it.
     - With an empty pad it returns the first `min(len, bytes.length)` bytes.
     - The default pad is the zero byte.
   - Type dispatch: `lpad(binary, n, string)` and `lpad(string, n, binary)` 
both use the string form and return STRING. Only (binary, n) and (binary, n, 
binary) return BINARY.
   - `len` comes in as INT after implicit cast. `lpad('hi', 'invalid_length')` 
fails with `CAST_INVALID_INPUT` under ANSI mode (the default since 4.0) and 
returns NULL with `spark.sql.ansi.enabled=false`. The cast itself is handled by 
the analyzer, but the kernel must accept Int32.
   - Very large results: the string form computes the result size with 
`Math.toIntExact` and throws `ArithmeticException` if it is over 2^31-1 bytes.
   - Scalar and array combinations for each argument (column `str` with literal 
`pad`, literal `str` with column `pad`, column `len`).
   - Collations (4.0+): any collation is accepted 
(`StringTypeWithCollation(supportsTrimCollation = true)`), but padding ignores 
collation. The result keeps the collation of `str`. Mixing two different 
explicit collations is an analysis error.
   
   ### Differences across Spark versions
   
   - 3.4 / 3.5: semantics as above. Binary-aware `lpad` already exists (added 
in 3.3.0).
   - 4.0: collation support added (`StringTypeWithCollation`). ANSI mode is on 
by default, which affects how a non-numeric `len` is cast. Padding semantics 
are unchanged for `UTF8_BINARY`.
   - 4.1 / 4.2: none found.
   
   ### Implementation notes
   
   - Spark source: 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala`
 (`LPadExpressionBuilder`, `PadExpressionBuilderBase`, `StringLPad`, 
`BinaryPad`). Kernels are in 
`common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java` 
(`lpad`) and 
`common/unsafe/src/main/java/org/apache/spark/unsafe/types/ByteArray.java` 
(`lpad`, `padWithEmptyPattern`, `fillWithPattern`).
   - DataFusion core has `lpad` (`datafusion/functions/src/unicode/lpad.rs`). 
Since apache/datafusion#21405 (DataFusion 54) it counts code points, not 
graphemes. Its string semantics are close to Spark's:
     - negative length becomes 0;
     - it truncates to the first `len` characters;
     - an empty fill returns the input unchanged.
   
     Remaining differences:
     - It has no BINARY support at all.
     - `len` is Int64, and it errors when `len > i32::MAX`. That can't happen 
in Spark because `len` is INT.
   
     A Spark wrapper could coerce `len` to Int32 semantics and delegate the 
string path to core, but it needs its own binary kernel.
   - Belongs in `crates/spark/src/function/string/` (e.g. a shared `pad.rs` 
with `rpad`). The binary kernel should share code with `rpad`.
   - Tests: there is already a placeholder 
`crates/sqllogictest/test_files/string/lpad.slt` with commented-out queries. 
Enable those and add the binary, negative-length, empty-pad and Unicode cases 
above. Document with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to