zyp-V commented on code in PR #65116:
URL: https://github.com/apache/doris/pull/65116#discussion_r3534218276
##########
be/src/core/data_type_serde/data_type_number_serde.cpp:
##########
@@ -867,22 +875,30 @@ void
DataTypeNumberSerDe<T>::write_one_cell_to_jsonb(const IColumn& column,
// both unsigned integers in Doris types and the JSONB types.
if constexpr (T == TYPE_TINYINT || T == TYPE_BOOLEAN) {
int8_t val = *reinterpret_cast<const int8_t*>(data_ref.data);
- result.writeInt8(val);
+ result.writeInt(val);
} else if constexpr (T == TYPE_SMALLINT) {
int16_t val = *reinterpret_cast<const int16_t*>(data_ref.data);
- result.writeInt16(val);
- } else if constexpr (T == TYPE_INT || T == TYPE_DATEV2 || T == TYPE_IPV4) {
+ result.writeInt(val);
+ } else if constexpr (T == TYPE_INT || T == TYPE_DATEV2) {
+ int32_t val = *reinterpret_cast<const int32_t*>(data_ref.data);
+ result.writeInt(val);
+ } else if constexpr (T == TYPE_IPV4) {
int32_t val = *reinterpret_cast<const int32_t*>(data_ref.data);
result.writeInt32(val);
} else if constexpr (T == TYPE_BIGINT || T == TYPE_DATE || T ==
TYPE_DATETIME ||
T == TYPE_DATETIMEV2 || T == TYPE_TIMESTAMPTZ) {
int64_t val = *reinterpret_cast<const int64_t*>(data_ref.data);
- result.writeInt64(val);
+ result.writeInt(val);
Review Comment:
Old readers cannot safely read newly written compact row-store JSONB because
the old code assumes an exact payload width for each Doris type.
For example, before the change a BIGINT value was always written as:
T_Int64 + 8-byte payload
So the old reader reads it with:
arg->unpack<JsonbInt64Val>()->val()
JsonbValue::unpack<T>() is just a raw reinterpret_cast of the payload. It
does not check the JSONB tag width or convert between integer types.
After the compact encoding change, the new writer may encode BIGINT 1 as:
T_Int8 + 1-byte payload
An old reader still treats that payload as JsonbInt64Val, so it reads 8
bytes from a 1-byte payload. That can misinterpret the value, read past the
valid payload, or produce undefined behavior.
However, we can add a BE config so that the new encoding is used only when
it is enabled, while both enabled and disabled states can still read existing
data correctly. Once data has been written with the new encoding, rollback to
code that does not support this feature is not safe. Other encoding
optimizations
follow the same pattern, such as enable_set_in_bitmap_value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
