amalsaidov opened a new pull request, #19271:
URL: https://github.com/apache/iotdb/pull/19271
## Description
`Binary` in `udf-api` and `pipe-api` compared its bytes with a signed
subtraction:
```java
return this.values[i] - other.values[i];
```
A Java `byte` is signed, so any byte with the high bit set (`>= 0x80`) is
negative. That makes `compareTo` sort such bytes before plain ASCII bytes,
which disagrees with:
- TsFile's own `org.apache.tsfile.utils.Binary.compareTo`, which compares
bytes unsigned (`val[index] & 0xff`), and
- `java.lang.String` ordering.
Any UDF/UDAF or pipe plugin that sorts or compares `Binary` values
containing non-ASCII UTF-8 text or binary data with high bytes therefore got
the wrong order.
### Scenario and before/after
`"A"` is the byte `0x41` (65). `"é"` is UTF-8 `0xC3 0xA9`, so its first byte
is `0xC3` (195 unsigned, but `-61` signed).
| Comparison | Correct (unsigned, String/TsFile) | Before this fix (signed) |
| --- | --- | --- |
| `new Binary("A").compareTo(new Binary("é"))` | `< 0` (A sorts first) | `>
0` (wrong) |
| `new Binary(new byte[]{0x41}).compareTo(new Binary(new
byte[]{(byte)0x80}))` | `< 0` | `> 0` (wrong) |
### Fix
Compare each byte as an unsigned value, matching the ordering used
everywhere else:
```java
return (this.values[i] & 0xFF) - (other.values[i] & 0xFF);
```
ASCII-only comparisons are unchanged (all ASCII bytes are `< 0x80`), and
`equals`, which only checks for `compareTo == 0`, is unaffected.
### Tests
Added `BinaryTest` to both modules covering unsigned ordering, agreement
with `String` ordering for non-ASCII text, ASCII ordering, and prefix/length
handling. The unsigned-ordering and String-agreement cases fail on the old code
and pass with the fix.
```
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 (udf-api BinaryTest)
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 (pipe-api BinaryTest)
```
This PR has:
- [x] been self-reviewed.
- [x] added unit tests to cover the fixed code path.
---
AI-assisted; directed, reviewed and tested by the author (amalsaidov).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]