amalsaidov opened a new pull request, #19271:
URL: https://github.com/apache/iotdb/pull/19271

   ## Description
   
   `Binary` in `udf-api` and `pipe-api` compared its bytes with a signed 
subtraction:
   
   ```java
   return this.values[i] - other.values[i];
   ```
   
   A Java `byte` is signed, so any byte with the high bit set (`>= 0x80`) is 
negative. That makes `compareTo` sort such bytes before plain ASCII bytes, 
which disagrees with:
   
   - TsFile's own `org.apache.tsfile.utils.Binary.compareTo`, which compares 
bytes unsigned (`val[index] & 0xff`), and
   - `java.lang.String` ordering.
   
   Any UDF/UDAF or pipe plugin that sorts or compares `Binary` values 
containing non-ASCII UTF-8 text or binary data with high bytes therefore got 
the wrong order.
   
   ### Scenario and before/after
   
   `"A"` is the byte `0x41` (65). `"é"` is UTF-8 `0xC3 0xA9`, so its first byte 
is `0xC3` (195 unsigned, but `-61` signed).
   
   | Comparison | Correct (unsigned, String/TsFile) | Before this fix (signed) |
   | --- | --- | --- |
   | `new Binary("A").compareTo(new Binary("é"))` | `< 0` (A sorts first) | `> 
0` (wrong) |
   | `new Binary(new byte[]{0x41}).compareTo(new Binary(new 
byte[]{(byte)0x80}))` | `< 0` | `> 0` (wrong) |
   
   ### Fix
   
   Compare each byte as an unsigned value, matching the ordering used 
everywhere else:
   
   ```java
   return (this.values[i] & 0xFF) - (other.values[i] & 0xFF);
   ```
   
   ASCII-only comparisons are unchanged (all ASCII bytes are `< 0x80`), and 
`equals`, which only checks for `compareTo == 0`, is unaffected.
   
   ### Tests
   
   Added `BinaryTest` to both modules covering unsigned ordering, agreement 
with `String` ordering for non-ASCII text, ASCII ordering, and prefix/length 
handling. The unsigned-ordering and String-agreement cases fail on the old code 
and pass with the fix.
   
   ```
   Tests run: 4, Failures: 0, Errors: 0, Skipped: 0  (udf-api BinaryTest)
   Tests run: 4, Failures: 0, Errors: 0, Skipped: 0  (pipe-api BinaryTest)
   ```
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added unit tests to cover the fixed code path.
   
   ---
   
   AI-assisted; directed, reviewed and tested by the author (amalsaidov).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to