This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new ab963392b1e [fix](be) Convert an encoded Variant UUID to STRING like a
typed one (#68938)
ab963392b1e is described below
commit ab963392b1ef3c5cff506a24bb4064848e2ece97
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Tue Oct 13 15:09:48 2026 +0800
[fix](be) Convert an encoded Variant UUID to STRING like a typed one
(#68938)
### What problem does this PR solve?
Issue Number: None
Related PR: #65561, #67627, #68935
Problem Summary:
**In short.** A Variant holds a UUID either as a typed value or as an
encoded Parquet Variant value, and `CAST(... AS STRING)` gives different
text for the two: the typed one becomes
`00112233-4455-6677-8899-aabbccddeeff`, the encoded one becomes the JSON
string `"00112233-4455-6677-8899-aabbccddeeff"`, quotes included. This
PR makes the encoded form convert like the typed one. It is not a
branch-4.x port (branch-4.x has no UUID type); it came up while testing
#68935.
**Background.** `ColumnVariantV2` keeps a column either typed (one Doris
scalar type) or encoded (Parquet Variant bytes), and the two states are
meant to give the same results; #67627 already checks that a typed and
an encoded UUID hash the same. For a STRING target,
`cast_variant_to_string.cpp` sends each encoded scalar either through
the concrete cast of its Doris type or to JSON text. #65561 wrote that
split before Doris had a UUID type, so `VariantPrimitiveId::UUID` went
to the JSON side together with BINARY and TIME_NTZ, which still have no
Doris type. #67627 then added the UUID type and the encoded-UUID group
in `cast_variant_to_scalar.cpp` (so `CAST(... AS UUID)` of an encoded
value works), but left the STRING split unchanged.
**The problem, and what it cost.** An encoded UUID shows up as soon as
the value goes through an encoded Variant: an element of a Variant
array, or a Paimon VARIANT column read back.
| Expression | master | with this PR |
|---|---|---|
| `CAST(CAST(CAST(u AS UUID) AS VARIANT) AS STRING)` (typed) |
`00112233-…` | `00112233-…` |
| `CAST(CAST(ARRAY(CAST(u AS UUID)) AS VARIANT)[1] AS STRING)` (encoded)
| `"00112233-…"` | `00112233-…` |
| `CAST(CAST(ARRAY(CAST(u AS UUID)) AS VARIANT) AS ARRAY<STRING>)` |
`[""00112233-…""]`: the element itself carries the quotes |
`["00112233-…"]` |
| `CAST(v AS STRING)` on a Paimon VARIANT column holding a UUID |
`"6ba7b810-…"` | `6ba7b810-…` |
| `CAST(CAST(ARRAY(CAST(u AS UUID)) AS VARIANT)[1] AS JSON)` |
`"00112233-…"` | unchanged (a JSON string) |
So the same value printed differently depending on how it was stored,
and a comparison such as `CAST(v AS STRING) = '00112233-…'` failed for
the encoded form. DATE, DECIMAL and TIMESTAMP values were not affected:
they already used their concrete cast.
**How this PR fixes it.** `uses_concrete_string_cast` now sends
`VariantPrimitiveId::UUID` to the concrete cast, which decodes the value
into the UUID group and casts it with the UUID-to-STRING cast, the same
path a typed UUID takes. BINARY and TIME_NTZ stay JSON text. `CAST(...
AS ARRAY<STRING>)` uses the same rule for its elements, so it changes
too; JSON output is a separate path and keeps the quoted string.
**Results.** The encoded and typed forms now give the same STRING, in a
whole-column cast, in a column that mixes UUIDs with JSON-only values,
and as `ARRAY<STRING>` elements.
---
.../cast/variant_v2/cast_variant_to_string.cpp | 2 +-
.../function/cast/cast_variant_v2_from_test.cpp | 91 +++++++++++++++++++---
.../uuid/test_uuid_variant_identity.out | 3 +
.../uuid/test_uuid_variant_identity.groovy | 10 +++
4 files changed, 94 insertions(+), 12 deletions(-)
diff --git a/be/src/exprs/function/cast/variant_v2/cast_variant_to_string.cpp
b/be/src/exprs/function/cast/variant_v2/cast_variant_to_string.cpp
index 069e58fc2c1..e218cc4e438 100644
--- a/be/src/exprs/function/cast/variant_v2/cast_variant_to_string.cpp
+++ b/be/src/exprs/function/cast/variant_v2/cast_variant_to_string.cpp
@@ -55,11 +55,11 @@ bool uses_concrete_string_cast(VariantRef value) {
case VariantPrimitiveId::STRING:
case VariantPrimitiveId::TIMESTAMP_NANOS:
case VariantPrimitiveId::TIMESTAMP_NTZ_NANOS:
+ case VariantPrimitiveId::UUID:
return true;
case VariantPrimitiveId::NULL_VALUE:
case VariantPrimitiveId::BINARY:
case VariantPrimitiveId::TIME_NTZ_MICROS:
- case VariantPrimitiveId::UUID:
return false;
}
throw Exception(ErrorCode::CORRUPTION, "Unknown Variant primitive id");
diff --git a/be/test/exprs/function/cast/cast_variant_v2_from_test.cpp
b/be/test/exprs/function/cast/cast_variant_v2_from_test.cpp
index 8ddbfc88bf4..69089d94058 100644
--- a/be/test/exprs/function/cast/cast_variant_v2_from_test.cpp
+++ b/be/test/exprs/function/cast/cast_variant_v2_from_test.cpp
@@ -41,6 +41,7 @@
#include "core/data_type/data_type_uuid.h"
#include "core/data_type/data_type_variant_v2.h"
#include "core/field.h"
+#include "core/value/uuid_value.h"
#include "core/value/variant/variant_batch_builder.h"
#include "core/value/vdatetime_value.h"
#include "exprs/function/cast/variant_v2/cast_variant_v2.h"
@@ -416,11 +417,7 @@ TEST(CastVariantV2FromTest,
TypedStringValidatesOnlyVisibleDateRows) {
TEST(CastVariantV2FromTest, EOnlyScalarStringRulesStayJsonQuoted) {
const std::array<char, 2> binary {0, 1};
- std::array<uint8_t, 16> uuid {};
- for (size_t index = 0; index < uuid.size(); ++index) {
- uuid[index] = static_cast<uint8_t>(index);
- }
- VariantBatchBuilder builder(VariantBatchBuilder::ReserveHint {.rows = 3});
+ VariantBatchBuilder builder(VariantBatchBuilder::ReserveHint {.rows = 2});
{
auto row = builder.begin_row();
row.add_binary(StringRef(binary.data(), binary.size()));
@@ -431,18 +428,12 @@ TEST(CastVariantV2FromTest,
EOnlyScalarStringRulesStayJsonQuoted) {
row.add_time_ntz_micros(1);
row.finish();
}
- {
- auto row = builder.begin_row();
- row.add_uuid(uuid);
- row.finish();
- }
CastResult cast = execute_from_variant(finish(&builder),
std::make_shared<DataTypeString>());
ASSERT_TRUE(cast.status.ok()) << cast.status;
const auto& nullable = nullable_result(cast.column);
const auto& strings = assert_cast<const
ColumnString&>(nullable.get_nested_column());
EXPECT_EQ(strings.get_data_at(0), StringRef(R"("AAE=")"));
EXPECT_EQ(strings.get_data_at(1), StringRef(R"("00:00:00.000001")"));
- EXPECT_EQ(strings.get_data_at(2),
StringRef(R"("00010203-0405-0607-0809-0a0b0c0d0e0f")"));
EXPECT_FALSE(nullable.has_null());
}
@@ -878,4 +869,82 @@ TEST(CastVariantV2FromTest,
EncodedUuidPreservesAllBitsAndNulls) {
EXPECT_TRUE(nullable_result(cast.column).is_null_at(0));
}
+TEST(CastVariantV2FromTest, EncodedUuidStringMatchesTypedUuidString) {
+ // UUID has a Doris type, so an encoded UUID converts to STRING through
the concrete UUID cast,
+ // as a typed UUID does, instead of keeping the JSON text of scalars that
have no Doris type.
+ const std::array<uint8_t, 16> bytes {0x00, 0x11, 0x22, 0x33, 0x44, 0x55,
0x66, 0x77,
+ 0x88, 0x99, 0xaa, 0xbb, 0xcc, 0xdd,
0xee, 0xff};
+ const StringRef expected("00112233-4455-6677-8899-aabbccddeeff");
+ const auto string_type = std::make_shared<DataTypeString>();
+
+ auto values = ColumnUUID::create();
+ values->insert_value(UUIDValue::from_big_endian(bytes.data()));
+ ColumnPtr typed = ColumnVariantV2::create_typed(
+ ColumnNullable::create(std::move(values), ColumnUInt8::create(1,
0)),
+ std::make_shared<DataTypeUUID>());
+ CastResult cast = execute_from_variant(typed, string_type);
+ ASSERT_TRUE(cast.status.ok()) << cast.status;
+ EXPECT_EQ(nullable_result(cast.column).get_nested_column().get_data_at(0),
expected);
+
+ // An all-UUID column takes the whole-column concrete cast.
+ VariantBatchBuilder uuid_only;
+ {
+ auto row = uuid_only.begin_row();
+ row.add_uuid(bytes);
+ row.finish();
+ }
+ cast = execute_from_variant(finish(&uuid_only), string_type);
+ ASSERT_TRUE(cast.status.ok()) << cast.status;
+ EXPECT_EQ(nullable_result(cast.column).get_nested_column().get_data_at(0),
expected);
+
+ // A mixed column puts the concrete UUID rows back in order with the JSON
fallback rows.
+ VariantBatchBuilder mixed(VariantBatchBuilder::ReserveHint {.rows = 3});
+ {
+ auto row = mixed.begin_row();
+ auto object = row.start_object();
+ object.add_key(StringRef("a"));
+ row.add_int(1);
+ object.finish();
+ row.finish();
+ }
+ {
+ auto row = mixed.begin_row();
+ row.add_uuid(bytes);
+ row.finish();
+ }
+ {
+ auto row = mixed.begin_row();
+ row.add_null();
+ row.finish();
+ }
+ cast = execute_from_variant(finish(&mixed), string_type);
+ ASSERT_TRUE(cast.status.ok()) << cast.status;
+ const auto& nullable = nullable_result(cast.column);
+ const auto& strings = assert_cast<const
ColumnString&>(nullable.get_nested_column());
+ ASSERT_EQ(strings.size(), 3);
+ EXPECT_EQ(strings.get_data_at(0), StringRef(R"({"a":1})"));
+ EXPECT_EQ(strings.get_data_at(1), expected);
+ EXPECT_EQ(strings.get_data_at(2), StringRef("null"));
+ EXPECT_FALSE(nullable.has_null());
+
+ // ARRAY<STRING> converts its elements with the same rule.
+ VariantBatchBuilder array_of_uuid;
+ {
+ auto row = array_of_uuid.begin_row();
+ auto array = row.start_array();
+ row.add_uuid(bytes);
+ array.finish();
+ row.finish();
+ }
+ cast = execute_from_variant(finish(&array_of_uuid),
+ std::make_shared<DataTypeArray>(string_type));
+ ASSERT_TRUE(cast.status.ok()) << cast.status;
+ const auto& arrays =
+ assert_cast<const
ColumnArray&>(nullable_result(cast.column).get_nested_column());
+ ASSERT_EQ(arrays.size_at(0), 1);
+ const auto& elements = assert_cast<const
ColumnNullable&>(arrays.get_data());
+ EXPECT_EQ(assert_cast<const
ColumnString&>(elements.get_nested_column()).get_data_at(0),
+ expected);
+}
+
} // namespace doris::CastWrapper
diff --git
a/regression-test/data/datatype_p0/uuid/test_uuid_variant_identity.out
b/regression-test/data/datatype_p0/uuid/test_uuid_variant_identity.out
index eb2e5a2ae30..8b862300b91 100644
--- a/regression-test/data/datatype_p0/uuid/test_uuid_variant_identity.out
+++ b/regression-test/data/datatype_p0/uuid/test_uuid_variant_identity.out
@@ -2,6 +2,9 @@
-- !native_uuid_variant_type --
uuid 00112233-4455-6677-8899-aabbccddeeff \N
+-- !uuid_variant_to_string --
+00112233-4455-6677-8899-aabbccddeeff 00112233-4455-6677-8899-aabbccddeeff
["00112233-4455-6677-8899-aabbccddeeff"]
"00112233-4455-6677-8899-aabbccddeeff" ["00112233-4455-6677-8899-aabbccddeeff"]
+
-- !persisted_uuid_variant_type --
1 uuid 00112233-4455-6677-8899-aabbccddeeff
2 \N \N
diff --git
a/regression-test/suites/datatype_p0/uuid/test_uuid_variant_identity.groovy
b/regression-test/suites/datatype_p0/uuid/test_uuid_variant_identity.groovy
index b246e9a9fc6..78167b65dec 100644
--- a/regression-test/suites/datatype_p0/uuid/test_uuid_variant_identity.groovy
+++ b/regression-test/suites/datatype_p0/uuid/test_uuid_variant_identity.groovy
@@ -21,6 +21,16 @@ suite("test_uuid_variant_identity") {
CAST(CAST(CAST('00112233-4455-6677-8899-aabbccddeeff' AS UUID)
AS VARIANT) AS UUID),
variant_type(CAST(CAST(NULL AS UUID) AS VARIANT))
"""
+ // An array element of a Variant is encoded, while a cast scalar is typed.
Both convert a UUID
+ // to the same STRING, without JSON quotes, also as an ARRAY<STRING>
element; JSON output
+ // keeps the quoted string.
+ qt_uuid_variant_to_string """
+ SELECT CAST(CAST(CAST('00112233-4455-6677-8899-aabbccddeeff' AS UUID)
AS VARIANT) AS STRING),
+ CAST(CAST(ARRAY(CAST('00112233-4455-6677-8899-aabbccddeeff' AS
UUID)) AS VARIANT)[1] AS STRING),
+ CAST(CAST(ARRAY(CAST('00112233-4455-6677-8899-aabbccddeeff' AS
UUID)) AS VARIANT) AS ARRAY<STRING>),
+ CAST(CAST(ARRAY(CAST('00112233-4455-6677-8899-aabbccddeeff' AS
UUID)) AS VARIANT)[1] AS JSON),
+ CAST(CAST(ARRAY(CAST('00112233-4455-6677-8899-aabbccddeeff' AS
UUID)) AS VARIANT) AS STRING)
+ """
sql "DROP TABLE IF EXISTS uuid_variant_identity"
sql """
CREATE TABLE uuid_variant_identity (
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
