github-actions[bot] commented on code in PR #69723:
URL: https://github.com/apache/doris/pull/69723#discussion_r4430592031
##########
be/src/format_v2/orc/orc_reader.cpp:
##########
@@ -907,6 +1168,14 @@ void OrcReader::_init_profile() {
Review Comment:
[P2] Keep ORC and Parquet footer counters distinct in a shared scanner
profile. This adds `FileFooterReadCalls` and `FileFooterHitCache` under
`OrcReader`, but `ParquetProfile` registers the same flat names under
`ParquetReader`, and `RuntimeProfile::add_counter` returns the existing counter
when the name is already present. A mixed-format scan therefore assigns the
second reader's footer reads and hits to whichever format initialized first and
merges the deltas into that subtree. Use format-qualified counter names or
explicit per-format children, and cover both initialization orders with a mixed
ORC/Parquet profile test.
##########
be/src/load/delta_writer/push_handler.cpp:
##########
@@ -380,171 +407,81 @@ Status PushBrokerReader::init() {
fmt::format("PushBrokerReader:dummy_id={}", print_id(dummy_id)));
_runtime_state = RuntimeState::create_unique(params, query_options,
query_globals,
ExecEnv::GetInstance(),
nullptr, tracker);
+ TDescriptorTable projected_descriptors;
+ RETURN_IF_ERROR(project_broker_source_slots(_t_desc_tbl, _params,
&projected_descriptors));
DescriptorTbl* desc_tbl = nullptr;
- Status status = DescriptorTbl::create(_runtime_state->obj_pool(),
_t_desc_tbl, &desc_tbl);
+ Status status =
+ DescriptorTbl::create(_runtime_state->obj_pool(),
projected_descriptors, &desc_tbl);
if (UNLIKELY(!status.ok())) {
return Status::Error<PUSH_INIT_ERROR>("Failed to create descriptor
table, msg: {}", status);
}
_runtime_state->set_desc_tbl(desc_tbl);
_runtime_profile = _runtime_state->runtime_profile();
_runtime_profile->set_name("PushBrokerReader");
- _file_cache_statistics.reset(new io::FileCacheStatistics());
- _file_reader_stats.reset(new io::FileReaderStats());
- _io_ctx.reset(new io::IOContext());
+ _file_cache_statistics = std::make_unique<io::FileCacheStatistics>();
+ _file_reader_stats = std::make_unique<io::FileReaderStats>();
+ _io_ctx = std::make_shared<io::IOContext>();
_io_ctx->file_cache_stats = _file_cache_statistics.get();
_io_ctx->file_reader_stats = _file_reader_stats.get();
_io_ctx->query_id = &_runtime_state->query_id();
- auto slot_descs = desc_tbl->get_tuple_descriptor(0)->slots();
- uint32_t idx = 0;
- for (auto& slot_desc : slot_descs) {
- _all_col_names.push_back(to_lower((slot_desc->col_name())));
- _col_name_to_block_idx.insert({to_lower(slot_desc->col_name()),
idx++});
+ auto* source = desc_tbl->get_tuple_descriptor(_params.src_tuple_id);
+ _dest_tuple_desc = desc_tbl->get_tuple_descriptor(_params.dest_tuple_id);
+ if (source == nullptr || _dest_tuple_desc == nullptr) {
+ return Status::InternalError("Unknown source or destination tuple in
push load");
}
-
Review Comment:
[P1] Derive the omitted broker file-column count without consuming path
slots. When `num_of_columns_from_file` is absent, this fallback sets it to all
`src_slot_ids`. A broker payload can include a path or partition slot in that
list, for example `{raw_value, path_value}` with `columns_from_path={"route"}`.
`LoadTableReader` then marks both slots as physical, never adds the path slot
to `_partition_positions`, and asks Parquet for a nonexistent `path_value`
column. The load can fail mapping/default validation or materialize the wrong
value instead of the route constant. Derive the physical count from the
path-slot metadata and add an omitted-count regression next to the existing
explicit-count case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
