Gabriel39 commented on code in PR #69723:
URL: https://github.com/apache/doris/pull/69723#discussion_r4430817282
##########
be/src/format_v2/orc/orc_reader.cpp:
##########
@@ -907,6 +1168,14 @@ void OrcReader::_init_profile() {
Review Comment:
Fixed in ea9057f8ad0. ORC now registers OrcFileFooterReadCalls and
OrcFileFooterHitCache, keeping Parquet's existing names unchanged. Both
initialization-order tests now check distinct counter pointers, independent
read/hit deltas and the format parent maps. Both tests failed before the fix
and pass afterward.
Validation: 311 related BE ASAN tests passed; clang-format 16 on all 60
affected C/C++ files, build hygiene and clang-tidy passed.
##########
be/src/exec/operator/file_scan_operator.cpp:
##########
@@ -245,41 +138,12 @@ Status
FileScanLocalState::_init_scanners(std::list<ScannerSPtr>* scanners) {
scan_params = _split_source->get_params();
}
RETURN_IF_ERROR(validate_file_scan_support(*scan_params));
- const bool is_load =
-
state()->desc_tbl().get_tuple_descriptor(scan_params->src_tuple_id) != nullptr;
- // TODO: Use scanner v2 for all queries.
- const bool use_file_scanner_v2 =
- should_use_file_scanner_v2(state()->query_options(), is_load,
*scan_params);
- _operator_profile->add_info_string("UseScannerV2", use_file_scanner_v2 ?
"true" : "false");
- const auto* output_tuple_desc =
state()->desc_tbl().get_tuple_descriptor(_output_tuple_id);
- DORIS_CHECK(output_tuple_desc != nullptr);
- const bool metadata_only_count =
- is_count_star_pushdown() &&
_split_source->all_ranges_have_table_level_row_count();
- if (!is_load && !use_file_scanner_v2 && !metadata_only_count &&
- std::ranges::any_of(output_tuple_desc->slots(), [](const
SlotDescriptor* slot) {
- return contains_variant_type(slot->type());
- })) {
- // A syntactic COUNT(*) alone is insufficient: every assigned range
must prove that the
- // legacy scanner will emit metadata counts without decoding a Variant
carrier.
- return Status::NotSupported(
- "External VARIANT columns require FileScannerV2; the legacy
file scanner does "
- "not support VARIANT");
- }
- if (use_file_scanner_v2) {
- _parquet_profile_cache =
-
std::make_unique<format::parquet::ParquetProfileCache>(_scanner_profile.get());
- }
+ _parquet_profile_cache =
+
std::make_unique<format::parquet::ParquetProfileCache>(_scanner_profile.get());
for (int i = 0; i < _max_scanners; ++i) {
- ScannerSPtr scanner;
- if (use_file_scanner_v2) {
- scanner = FileScannerV2::create_shared(state(), this, p._limit,
_split_source,
- _scanner_profile.get(),
p._kv_cache.get(),
- &p._colname_to_slot_id);
- } else {
- scanner = FileScanner::create_shared(state(), this, p._limit,
_split_source,
- _scanner_profile.get(),
p._kv_cache.get(),
- &p._colname_to_slot_id);
- }
+ ScannerSPtr scanner = FileScannerV2::create_shared(
+ state(), this, p._limit, _split_source,
_scanner_profile.get(), p._kv_cache.get(),
Review Comment:
Follow-up in ea9057f8ad0: serialized-tail reuse remains in place, while the
new shared get_stable_key API and stricter ORC identity policy have been
reverted. ORC uses the existing V1 FileMetaCache::get_key policy, including its
file-size fallback when mtime is absent; the V2 payload discriminator remains.
The native Parquet key builder is restored unchanged. Cache-policy hardening is
outside this retirement change.
Warm opens, versioned keys, the legacy versionless fallback, disabled
caching and repeated close are covered. The ORC footer counters are
format-qualified to keep their ownership independent of Parquet in a shared
scanner profile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
