Gabriel39 commented on code in PR #67784:
URL: https://github.com/apache/doris/pull/67784#discussion_r4026319690
##########
be/src/format/arrow/arrow_block_convertor.h:
##########
@@ -43,32 +43,59 @@ class Schema;
namespace doris {
+// ORC and Arrow Iceberg writers share this parser so textual and binary UUID
inputs always use
+// the same canonical 16-byte representation.
Status parse_iceberg_uuid_to_bytes(StringRef uuid, std::array<uint8_t, 16>*
bytes);
+class ArrowWriteConverter {
Review Comment:
Addressed in 9731b5ed86. Refactored the conversion hierarchy around
ArrowBlockConvertor, with convert_from_arrow / convert_to_arrow and
ArrowFlightArrowBlockConvertor, IcebergArrowBlockConvertor, and
PaimonArrowBlockConvertor specializations. Iceberg schema translation is owned
by the Iceberg converter constructor, and the Parquet transformer uses that
converter directly. The overlapping converter classes were removed. Arrow
round-trip, sliced-array, null, UUID/fixed-binary, and writer coverage passed
locally.
##########
be/src/format_v2/table/paimon_reader.cpp:
##########
@@ -179,6 +180,47 @@ Status annotate_paimon_variants(const
std::vector<ColumnDefinition>& table_schem
return Status::OK();
}
+DataTypePtr nullable_like_original(const DataTypePtr& original, DataTypePtr
nested) {
+ return original != nullptr && original->is_nullable() ?
make_nullable(nested) : nested;
+}
+
+DataTypePtr apply_paimon_timestamp_semantics(format::ColumnDefinition* column)
{
+ DORIS_CHECK(column != nullptr);
+ DORIS_CHECK(column->type != nullptr);
+ const auto primitive = remove_nullable(column->type)->get_primitive_type();
+ if (column->timestamp_is_adjusted_to_utc.has_value() &&
+ (primitive == TYPE_DATETIMEV2 || primitive == TYPE_TIMESTAMPTZ)) {
+ const auto target =
Review Comment:
Addressed in 9731b5ed86. Fixed by checking the localized file predicates
before enabling the condition cache. Schema-evolution casts that cannot provide
a safe predicate snapshot now disable both cache reads and writes, so the
scanner-level digest cannot reuse an old all-false bitmap under changed
timestamp semantics. Added a regression with the same field ID and file
before/after TIMESTAMP-to-TIMESTAMP_LTZ evolution; it reproduced dropped rows
before the fix and passes afterward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
