github-actions[bot] commented on code in PR #70476:
URL: https://github.com/apache/doris/pull/70476#discussion_r4539270352
##########
be/src/format_v2/table/load_reader.cpp:
##########
@@ -99,6 +109,54 @@ Status LoadTableReader::init(TableReadOptions&& options) {
.condition_cache_digest = 0});
}
+void LoadTableReader::_init_parquet_variant_projections() {
+ // A native source slot can replace the planned text slot only if no other
expression reads
+ // it. For example, a pre-filter using string operations must still see
its planned type.
+ std::map<int, size_t> consumers;
+ for (const auto* expressions : {&_pre_filters, &_destination_exprs}) {
+ for (const auto& expression : *expressions) {
+ std::set<int> positions;
+ expression->root()->collect_slot_column_ids(positions);
+ for (int position : positions) {
+ ++consumers[position];
+ }
+ }
+ }
+
+ _parquet_variant_source_positions.resize(_destination_exprs.size(), -1);
+ for (size_t i = 0; i < _destination_exprs.size(); ++i) {
+ const int position = _direct_source_positions[i];
+ const auto& root = _destination_exprs[i]->root();
+ if (position < 0 || consumers[position] != 1 ||
Review Comment:
[P1] Preserve VARIANT values when the load source has another consumer
A load of the newly exported Parquet with direct `(id, v)` mapping and
`PRECEDING FILTER v IS NOT NULL` has two consumers of `v`, so this guard
disables native preservation. The mapper then casts the native VARIANT to the
planned VARCHAR using `EvolutionCast`/ordinary SQL CAST, and the destination
runs `try_parse_to_variant` on that text: root string `"true"` becomes BOOLEAN,
BOOLEAN `true` becomes integer `1`, and a root string `"hello"` becomes NULL or
a rejected row. Before this change, the UTF-8 export used JSON serialization
and all of these values survived the same load. Keep a lossless representation
for the VARIANT destination when another consumer needs text (for example, a
separate native value or JSON-preserving text view), and cover this export/load
case with a source filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
