pzhdfy commented on code in PR #69493:
URL: https://github.com/apache/doris/pull/69493#discussion_r4358682095
##########
regression-test/framework/src/main/groovy/org/apache/doris/regression/util/ResultUtils.groovy:
##########
@@ -84,6 +89,87 @@ class ResultUtils {
}
}
+ /**
+ * Brings nested binary values rendered by the two engines to a common
form.
+ *
+ * <p>Doris renders a VARBINARY nested in arrays/maps/structs as a
+ * "0x"-prefixed hex token ("[0x616C..., null]") while Spark's JDBC returns
+ * the raw bytes ("[alice-array-bin, null]"), so the same value compares
+ * unequal across engines. Decoding the tokens on both rows applies the
same
+ * substitution to both sides — a no-op on the raw rendering — so equal
stays
+ * equal, and scalar binary columns are unaffected: they arrive as byte[]
+ * and normalizeValue already hex-encodes them symmetrically.
+ */
+ static List<List<Object>> decodeNestedBinaryTokens(List<List<Object>>
rows) {
+ List<List<Object>> decodedRows = new ArrayList<>(rows.size())
+ for (List<Object> row : rows) {
+ List<Object> decodedRow = new ArrayList<>(row.size())
+ for (Object value : row) {
+ decodedRow.add(decodeNestedBinaryTokens(value))
+ }
+ decodedRows.add(decodedRow)
+ }
+ return decodedRows
+ }
+
+ static Object decodeNestedBinaryTokens(Object value) {
+ if (value instanceof String) {
+ String text = (String) value
+ Matcher matcher = NESTED_BINARY_TOKEN.matcher(text)
Review Comment:
Confirmed: the `0x`-token substitution ran as an unanchored substring
replacement on every normalized string. The pattern now requires a
value/element boundary (negative lookbehind on alphanumerics, quotes and
backslash), so only shapes Doris can only have produced as a binary element
(`"[0x616C, "`, `"{0x41: "`, `"ROW(0x61"`) decode — `prefix0x41`, quoted
strings and map-key fragments no longer collapse. The quantifier also no
longer matches a bare `0x` with zero byte pairs. A literal string whose whole
value coincides with a bare token ("0x41") remains the one documented
ambiguity; distinguishing it needs column types the comparison layer does not
have. Verified green against the spark-vs-doris oracle suites that exercise
the decode (`test_paimon_write_append_only`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
