This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 55f38f22789 branch-4.1: [fix](iceberg) Fix V2 reads across nested
schema evolution #67574 (#67690)
55f38f22789 is described below
commit 55f38f227896d429459f443806e79245babf8556
Author: Gabriel <[email protected]>
AuthorDate: Thu Sep 10 11:27:26 2026 +0800
branch-4.1: [fix](iceberg) Fix V2 reads across nested schema evolution
#67574 (#67690)
### What problem does this PR solve?
This PR backports #67574 to `branch-4.1`.
Two Iceberg V2 failures are fixed:
1. A query that projected two nested siblings returned `[10, 11]`
instead of `[11, 12]` after a predicate on a missing nested child was
rejected.
2. A migrated ID-less nested Parquet file returned stale physical values
where the Iceberg schema required `[NULL, ...]` after
`schema.name-mapping.default` was removed.
The first failure was caused by a stale nested file projection.
Predicate demotion widened the physical struct projection, but the table
mapping still used child ordinals from the earlier narrow projection.
The second failure occurred because an ID-less file could fall back to
current field names even when no authoritative Iceberg name mapping
remained.
### Behavior changes
Before this change:
- A rejected nested predicate could shift projected sibling values by
physical ordinal.
- V2 could match ID-less migrated fields by current names after the
explicit name mapping was removed, exposing stale data.
- Fuzzy session initialization could randomly select the legacy file
scanner.
After this change:
- Table mappings are reconciled with the final physical projection after
predicate demotion and table-format customization.
- V2 reads ID-less files by name only when an authoritative Iceberg
mapping exists, including mappings available only through schema
history.
- A present but malformed name-mapping property falls back to a mapping
generated from the current schema; an absent property remains
distinguishable and does not authorize current-name matching.
- Fuzzy sessions consistently initialize with File Scanner V2. Dedicated
compatibility tests can still explicitly select the legacy scanner
afterward.
The behavior change is limited to file-scanner selection in fuzzy tests
and Iceberg V2 schema-evolution reads. Queries that previously exposed
stale physical values can now return NULL, or a missing-required-field
error for required columns, which follows Iceberg field identity
semantics.
### Implementation
- Reapply finalized scan projections to column mappings and rebuild
localized filter entries.
- Compact file-block positions and remap localized slot references after
predicate demotion.
- Reconcile mappings after Iceberg adds hidden equality-delete
dependencies.
- Require authoritative name mapping for V2 ID-less files and inspect
complete schema history for metadata-only scans.
- Preserve current-schema fallback for malformed mapping properties in
the branch-4.1 `fe-core` Iceberg utility.
### Feature, refactoring, and optimization scope
- No new user-facing feature is added.
- Refactoring is limited to introducing the projection-reconciliation
step needed to preserve the scanner layout invariant.
- No performance optimization is intended; the additional work occurs
during scan-request construction rather than per-row processing.
### branch-4.1 adaptations
- Port the dense file-block position remapping dependency that already
exists in the original PR baseline.
- Apply the FE mapping fallback in the branch-4.1 `fe-core` Iceberg
utility because the newer connector module is not present on this
branch.
- Validate metadata-only mapping through schema history directly because
branch-4.1 does not expose synthesized table columns.
### Test
- ASAN BE unit-test build passed.
- `ColumnMapperTest.*` and `IcebergV2ReaderTest.*`: 102 tests passed.
-
`IcebergUtilsTest#testMalformedNameMappingFallsBackToCurrentSchemaNames`
passed.
- `SessionVariablesTest#testFileScannerV2StaysEnabledInFuzzyMode`
passed.
- FE Checkstyle passed.
- clang-format 16 passed for every affected C/C++ file.
- External regression coverage is included for nested sibling projection
alignment, migrated ID-less nested NULL semantics, and missing required
fields.
---
be/src/format_v2/column_mapper.cpp | 107 +++++++++
be/src/format_v2/column_mapper.h | 4 +
be/src/format_v2/table/iceberg_reader.cpp | 71 ++++++
be/src/format_v2/table/iceberg_reader.h | 8 +
be/src/format_v2/table/iceberg_schema_utils.h | 12 +
be/src/format_v2/table_reader.cpp | 2 +
be/src/format_v2/table_reader.h | 4 +-
be/test/format_v2/column_mapper_test.cpp | 38 ++++
be/test/format_v2/table/iceberg_reader_test.cpp | 251 ++++++++++++++++++++-
.../iceberg_load/run01.sql | 10 +-
.../data/data.parquet | Bin 0 -> 1081 bytes
...aafe0-9de1-4b3f-942d-305dc33fb82f.metadata.json | 1 +
...09fba-61c6-4e4b-b0f5-6023f70d5033.metadata.json | 1 +
...3fffe-54f9-486a-b15e-7333fdb96f61.metadata.json | 1 +
...3e871-e421-48ed-a735-354f4e1c9502.metadata.json | 1 +
...544-1-94f693c5-6efb-4e9d-a40e-f02ac301fc14.avro | Bin 0 -> 4667 bytes
...ifest-d04009d1-83e2-4d2a-859e-79d57e8dd381.avro | Bin 0 -> 7501 bytes
.../doris/datasource/iceberg/IcebergUtils.java | 8 +-
.../java/org/apache/doris/qe/SessionVariable.java | 3 +
.../doris/datasource/iceberg/IcebergUtilsTest.java | 17 ++
.../org/apache/doris/qe/SessionVariablesTest.java | 1 +
.../iceberg/test_gen_iceberg_by_api.out | 3 -
...ceberg_migrated_nested_without_name_mapping.out | 3 +
.../test_iceberg_struct_schema_evolution.out | 3 +
.../iceberg/test_gen_iceberg_by_api.groovy | 9 +-
...erg_migrated_nested_without_name_mapping.groovy | 55 +++++
.../test_iceberg_struct_schema_evolution.groovy | 9 +
27 files changed, 607 insertions(+), 15 deletions(-)
diff --git a/be/src/format_v2/column_mapper.cpp
b/be/src/format_v2/column_mapper.cpp
index 57132df997f..f123917f4b2 100644
--- a/be/src/format_v2/column_mapper.cpp
+++ b/be/src/format_v2/column_mapper.cpp
@@ -20,6 +20,7 @@
#include <algorithm>
#include <cctype>
#include <cstddef>
+#include <map>
#include <memory>
#include <optional>
#include <set>
@@ -47,6 +48,7 @@
#include "exprs/vexpr_context.h"
#include "exprs/vin_predicate.h"
#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
#include "format_v2/column_mapper_nested.h"
#include "format_v2/expr/cast.h"
#include "format_v2/file_reader.h"
@@ -243,6 +245,56 @@ std::string field_debug_string(const Field& field) {
return out.str();
}
+void remap_localized_slot_positions(const VExprSPtr& expr,
+ const std::map<size_t, size_t>&
position_remap,
+ std::set<const VExpr*>* visited) {
+ if (expr == nullptr || !visited->insert(expr.get()).second) {
+ return;
+ }
+ if (auto* slot = dynamic_cast<VSlotRef*>(expr.get());
+ slot != nullptr && slot->column_id() >= 0) {
+ const auto remap =
position_remap.find(static_cast<size_t>(slot->column_id()));
+ DORIS_CHECK(remap != position_remap.end()) << slot->column_id();
+ slot->set_column_id(static_cast<int>(remap->second));
+ }
+ for (const auto& child : expr->children()) {
+ remap_localized_slot_positions(child, position_remap, visited);
+ }
+ remap_localized_slot_positions(expr->get_impl(), position_remap, visited);
+}
+
+void compact_file_block_positions(FileScanRequest* request) {
+ std::set<size_t> occupied_positions;
+ for (const auto& [_, position] : request->local_positions) {
+ occupied_positions.insert(position.value());
+ }
+ for (const auto& [_, position] : request->non_predicate_positions) {
+ occupied_positions.insert(position.value());
+ }
+
+ std::map<size_t, size_t> position_remap;
+ size_t dense_position = 0;
+ for (size_t old_position : occupied_positions) {
+ position_remap.emplace(old_position, dense_position++);
+ }
+ for (auto& [_, position] : request->local_positions) {
+ position = LocalIndex(position_remap.at(position.value()));
+ }
+ for (auto& [_, position] : request->non_predicate_positions) {
+ position = LocalIndex(position_remap.at(position.value()));
+ }
+
+ // Slot refs were localized before predicate demotion removed duplicate
file-block positions;
+ // remap them with the compacted layout so residual predicates keep
reading the same columns.
+ std::set<const VExpr*> visited;
+ for (const auto& conjunct : request->conjuncts) {
+ remap_localized_slot_positions(conjunct->root(), position_remap,
&visited);
+ }
+ for (const auto& conjunct : request->delete_conjuncts) {
+ remap_localized_slot_positions(conjunct->root(), position_remap,
&visited);
+ }
+}
+
template <typename T, typename Formatter>
std::string join_debug_strings(const std::vector<T>& values, Formatter
formatter) {
std::ostringstream out;
@@ -2023,6 +2075,20 @@ static const LocalColumnIndex* find_scan_projection(
return projection_it == scan_columns.end() ? nullptr : &*projection_it;
}
+static bool same_projected_file_shape(const std::vector<ColumnDefinition>& lhs,
+ const std::vector<ColumnDefinition>&
rhs) {
+ if (lhs.size() != rhs.size()) {
+ return false;
+ }
+ for (size_t index = 0; index < lhs.size(); ++index) {
+ if (lhs[index].local_id != rhs[index].local_id ||
+ !same_projected_file_shape(lhs[index].children,
rhs[index].children)) {
+ return false;
+ }
+ }
+ return true;
+}
+
// Apply the final scan projection of one root file column back to its
ColumnMapping. This updates
// mapping.file_type/projected_file_children from the original file schema to
the exact shape that
// FileReader will return.
@@ -2428,6 +2494,36 @@ Status TableColumnMapper::create_scan_request(
return Status::OK();
}
+Status TableColumnMapper::reconcile_scan_request_after_customization(
+ FileScanRequest* file_request) {
+ DORIS_CHECK(file_request != nullptr);
+ bool output_shape_changed = false;
+ for (auto& mapping : _mappings) {
+ if (!mapping.file_local_id.has_value() ||
+
!file_request->local_positions.contains(LocalColumnId(*mapping.file_local_id)))
{
+ continue;
+ }
+ const auto previous_file_type = mapping.file_type;
+ const auto previous_file_children = mapping.projected_file_children;
+
RETURN_IF_ERROR(apply_scan_projection_to_mapping_file_type(*file_request,
&mapping));
+ output_shape_changed |=
+ previous_file_type == nullptr || mapping.file_type == nullptr
||
+ !previous_file_type->equals(*mapping.file_type) ||
+ !same_projected_file_shape(previous_file_children,
mapping.projected_file_children);
+ rebuild_projection(&mapping, file_request->non_predicate_position(
+
LocalColumnId(*mapping.file_local_id)));
+ }
+ if (output_shape_changed) {
+ // Localized conjuncts embed nested child ordinals from the pre-hook
projection. Scanner
+ // still evaluates the original table conjuncts, so discard stale
file-local copies rather
+ // than allowing a late equality-delete dependency to reinterpret
another child.
+ file_request->conjuncts.clear();
+ file_request->metadata_pruning_safe_conjunct_count = 0;
+ }
+ RETURN_IF_ERROR(_build_filter_entries(*file_request));
+ return Status::OK();
+}
+
ColumnMapping* TableColumnMapper::_find_mapping(GlobalIndex global_index) {
for (auto& mapping : _mappings) {
if (mapping.global_index == global_index) {
@@ -2682,6 +2778,17 @@ Status TableColumnMapper::localize_filters(const
std::vector<TableFilter>& table
FileScanRequestBuilder builder(file_request);
RETURN_IF_ERROR(builder.add_non_predicate_column(std::move(demoted_projection)));
}
+ // Predicate demotion can widen a nested projection after mappings were
localized. Reapply the
+ // final shape so TableReader interprets the same child ordinals that
FileReader returns.
+ for (auto& mapping : _mappings) {
+ if (mapping.file_local_id.has_value() &&
+
file_request->local_positions.contains(LocalColumnId(*mapping.file_local_id))) {
+
RETURN_IF_ERROR(apply_scan_projection_to_mapping_file_type(*file_request,
&mapping));
+ }
+ }
+ // Final readers allocate a dense file block, so every retained slot must
follow the same compaction.
+ compact_file_block_positions(file_request);
+ RETURN_IF_ERROR(_build_filter_entries(*file_request));
return Status::OK();
}
diff --git a/be/src/format_v2/column_mapper.h b/be/src/format_v2/column_mapper.h
index ccfbd090407..cd2959c8700 100644
--- a/be/src/format_v2/column_mapper.h
+++ b/be/src/format_v2/column_mapper.h
@@ -206,6 +206,10 @@ public:
RuntimeState* runtime_state = nullptr,
const std::map<LocalColumnId, LocalIndex>* fixed_local_positions =
nullptr);
+ // Table-format hooks may append hidden physical dependencies after the
initial request is
+ // localized. Reconcile output mappings with that final layout before
opening expressions.
+ Status reconcile_scan_request_after_customization(FileScanRequest*
file_request);
+
// Localize table-level filters to the file schema.
// Trivial mappings can copy structured predicates directly. Type changes
may be localized with
// a safe cast. Expressions that cannot be pushed down safely should be
handled by the
diff --git a/be/src/format_v2/table/iceberg_reader.cpp
b/be/src/format_v2/table/iceberg_reader.cpp
index 078b413a059..c18cec8108f 100644
--- a/be/src/format_v2/table/iceberg_reader.cpp
+++ b/be/src/format_v2/table/iceberg_reader.cpp
@@ -55,6 +55,7 @@
#include "format_v2/orc/orc_reader.h"
#include "format_v2/parquet/parquet_reader.h"
#include "format_v2/parquet/reader/column_reader.h"
+#include "format_v2/table/schema_history_util.h"
#include "format_v2/table_reader.h"
#include "io/file_factory.h"
#include "util/debug_points.h"
@@ -66,6 +67,76 @@ namespace doris::format::iceberg {
static constexpr const char* ROW_LINEAGE_ROW_ID = "_row_id";
static constexpr int32_t ROW_LINEAGE_ROW_ID_FIELD_ID = 2147483540;
+namespace {
+
+const schema::external::TField* get_external_field_ptr(
+ const schema::external::TFieldPtr& field_ptr) {
+ if (!field_ptr.__isset.field_ptr || field_ptr.field_ptr == nullptr) {
+ return nullptr;
+ }
+ return field_ptr.field_ptr.get();
+}
+
+bool external_field_has_authoritative_name_mapping(const
schema::external::TField& field) {
+ if (field.__isset.name_mapping_is_authoritative &&
field.name_mapping_is_authoritative) {
+ return true;
+ }
+ if (!field.__isset.nestedField) {
+ return false;
+ }
+ if (field.nestedField.__isset.struct_field &&
field.nestedField.struct_field.__isset.fields) {
+ return std::ranges::any_of(field.nestedField.struct_field.fields,
[](const auto& child) {
+ const auto* child_field = get_external_field_ptr(child);
+ return child_field != nullptr &&
+ external_field_has_authoritative_name_mapping(*child_field);
+ });
+ }
+ if (field.nestedField.__isset.array_field &&
field.nestedField.array_field.__isset.item_field) {
+ const auto* item =
get_external_field_ptr(field.nestedField.array_field.item_field);
+ return item != nullptr &&
external_field_has_authoritative_name_mapping(*item);
+ }
+ if (field.nestedField.__isset.map_field) {
+ const auto& map_field = field.nestedField.map_field;
+ if (map_field.__isset.key_field) {
+ const auto* key = get_external_field_ptr(map_field.key_field);
+ if (key != nullptr &&
external_field_has_authoritative_name_mapping(*key)) {
+ return true;
+ }
+ }
+ if (map_field.__isset.value_field) {
+ const auto* value = get_external_field_ptr(map_field.value_field);
+ return value != nullptr &&
external_field_has_authoritative_name_mapping(*value);
+ }
+ }
+ return false;
+}
+
+} // namespace
+
+bool IcebergTableReader::_scan_has_any_authoritative_name_mapping() const {
+ if (schema_has_any_authoritative_name_mapping(_projected_columns)) {
+ return true;
+ }
+ if (_scan_params == nullptr || !_scan_params->__isset.history_schema_info)
{
+ return false;
+ }
+ // Metadata-only scans have no projected data column carrying aliases.
Consult the complete
+ // schema so hidden equality-delete keys use the same authoritative
mapping as visible fields.
+ for (const auto& schema : _scan_params->history_schema_info) {
+ if (!schema.__isset.root_field || !schema.root_field.__isset.fields) {
+ continue;
+ }
+ if (std::ranges::any_of(schema.root_field.fields, [](const auto&
field) {
+ const auto* schema_field = get_external_field_ptr(field);
+ return schema_field != nullptr &&
+
external_field_has_authoritative_name_mapping(*schema_field);
+ })) {
+ return true;
+ }
+ }
+ return false;
+}
+
template <typename T>
static std::string join_values_for_debug(const std::vector<T>& values) {
std::ostringstream out;
diff --git a/be/src/format_v2/table/iceberg_reader.h
b/be/src/format_v2/table/iceberg_reader.h
index 5760631e577..21609b9866f 100644
--- a/be/src/format_v2/table/iceberg_reader.h
+++ b/be/src/format_v2/table/iceberg_reader.h
@@ -69,6 +69,12 @@ public:
if (!_data_reader.file_schema.empty() && has_field_ids) {
return format::TableColumnMappingMode::BY_FIELD_ID;
}
+ if (!_data_reader.file_schema.empty() &&
supports_iceberg_scan_semantics_v2(_scan_params) &&
+ !_scan_has_any_authoritative_name_mapping()) {
+ // ID-less migrated files are name-readable only while Iceberg's
explicit default name
+ // mapping exists; current names must not resurrect file fields
after it is removed.
+ return format::TableColumnMappingMode::BY_FIELD_ID;
+ }
return format::TableColumnMappingMode::BY_NAME;
}
@@ -113,6 +119,8 @@ private:
static constexpr size_t ICEBERG_FILE_PATH_BLOCK_POSITION = 0;
static constexpr size_t ICEBERG_ROW_POS_BLOCK_POSITION = 1;
+ bool _scan_has_any_authoritative_name_mapping() const;
+
class PositionDeleteRowsCollector final {
public:
using PositionDeleteFile = std::unordered_map<std::string,
format::DeleteRows>;
diff --git a/be/src/format_v2/table/iceberg_schema_utils.h
b/be/src/format_v2/table/iceberg_schema_utils.h
index 516c982194b..c37b907c018 100644
--- a/be/src/format_v2/table/iceberg_schema_utils.h
+++ b/be/src/format_v2/table/iceberg_schema_utils.h
@@ -50,4 +50,16 @@ inline bool schema_has_all_field_ids(const
std::vector<ColumnDefinition>& schema
return true;
}
+inline bool schema_has_any_authoritative_name_mapping(const
std::vector<ColumnDefinition>& schema) {
+ for (const auto& field : schema) {
+ if (field.column_type != ColumnType::DATA_COLUMN) {
+ continue;
+ }
+ if (field.has_name_mapping ||
schema_has_any_authoritative_name_mapping(field.children)) {
+ return true;
+ }
+ }
+ return false;
+}
+
} // namespace doris::format::iceberg
diff --git a/be/src/format_v2/table_reader.cpp
b/be/src/format_v2/table_reader.cpp
index 78089087c76..8e03d3d2282 100644
--- a/be/src/format_v2/table_reader.cpp
+++ b/be/src/format_v2/table_reader.cpp
@@ -1354,6 +1354,8 @@ Status TableReader::refresh_conjuncts(VExprContextSPtrs
conjuncts,
}
}
RETURN_IF_ERROR(customize_file_scan_request(refreshed_request.get()));
+ RETURN_IF_ERROR(
+
refreshed_mapper->reconcile_scan_request_after_customization(refreshed_request.get()));
if (_file_scan_request == nullptr ||
!same_physical_scan_layout(*refreshed_request, *_file_scan_request)) {
// A reader cannot reinterpret columns already materialized with
another block layout.
diff --git a/be/src/format_v2/table_reader.h b/be/src/format_v2/table_reader.h
index 0d6928b96b9..797bba2a0a1 100644
--- a/be/src/format_v2/table_reader.h
+++ b/be/src/format_v2/table_reader.h
@@ -487,7 +487,6 @@ protected:
RETURN_IF_ERROR(close_current_reader());
return Status::OK();
}
- RETURN_IF_ERROR(validate_file_mapping(*_data_reader.column_mapper));
// COUNT(*) has no semantic column argument, but Nereids retains a
minimum-width scan slot
// so the scan node still has an output tuple. Record only the current
non-predicate file
// columns before table-format hooks add row-position or
equality-delete dependencies. This
@@ -508,6 +507,9 @@ protected:
}
}
RETURN_IF_ERROR(customize_file_scan_request(file_request.get()));
+
RETURN_IF_ERROR(_data_reader.column_mapper->reconcile_scan_request_after_customization(
+ file_request.get()));
+ RETURN_IF_ERROR(validate_file_mapping(*_data_reader.column_mapper));
RETURN_IF_ERROR(_open_local_filter_exprs(*file_request));
_data_reader.file_block_layout.clear();
_data_reader.block_template.clear();
diff --git a/be/test/format_v2/column_mapper_test.cpp
b/be/test/format_v2/column_mapper_test.cpp
index 06bbd949396..ff402bd0502 100644
--- a/be/test/format_v2/column_mapper_test.cpp
+++ b/be/test/format_v2/column_mapper_test.cpp
@@ -4213,6 +4213,44 @@ TEST(ColumnMapperTest,
PredicateAccessPathsCreateDeferredStructOutputProjection)
EXPECT_TRUE(request.is_predicate_only(LocalColumnId(0)));
}
+TEST(ColumnMapperTest,
RejectedMissingStructPredicateRestoresFullOutputMapping) {
+ auto table_renamed = field_id_col("renamed", 2, i64());
+ auto table_keep = field_id_col("keep", 3, i64());
+ auto table_added = field_id_col("added", 6, i64());
+ auto table_struct = struct_col("s", 1, {table_renamed, table_keep});
+ auto full_table_struct = struct_col("s", 1, {table_renamed, table_keep,
table_added});
+ table_struct.type = full_table_struct.type;
+ table_struct.has_predicate_access_paths = true;
+ table_struct.predicate_children = {table_added};
+
+ auto file_removed = field_id_col("removed", 7, i64(), 0);
+ auto file_renamed = field_id_col("rename_me", 2, i64(), 1);
+ auto file_keep = field_id_col("keep", 3, i64(), 2);
+ auto file_struct = struct_col("s", 1, {file_removed, file_renamed,
file_keep}, 0);
+
+ ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID});
+ ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok());
+
+ auto added = struct_element(table_slot(0, 0, table_struct.type, "s"),
i64(), "added");
+ auto predicate = binary_predicate(TExprOpcode::GT, added,
+ literal(i64(),
Field::create_field<TYPE_BIGINT>(0)));
+ TableFilter filter {.conjunct = VExprContext::create_shared(predicate),
+ .global_indices = {GlobalIndex(0)}};
+
+ FileScanRequest request;
+ ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct},
&request).ok());
+ EXPECT_TRUE(request.predicate_columns.empty());
+ ASSERT_EQ(request.non_predicate_columns.size(), 1) <<
request.debug_string();
+ EXPECT_TRUE(request.non_predicate_columns[0].project_all_children);
+
+ ASSERT_EQ(mapper.mappings().size(), 1);
+ const auto& mapping = mapper.mappings()[0];
+ ASSERT_EQ(mapping.projected_file_children.size(), 3);
+ EXPECT_EQ(mapping.projected_file_children[0].name, "removed");
+ EXPECT_EQ(mapping.projected_file_children[1].name, "rename_me");
+ EXPECT_EQ(mapping.projected_file_children[2].name, "keep");
+}
+
TEST(ColumnMapperTest,
PredicateAccessPathsCreateDeferredVariantRootProjection) {
auto table_variant = field_id_col("v", 10, variant_v2());
table_variant.has_predicate_access_paths = true;
diff --git a/be/test/format_v2/table/iceberg_reader_test.cpp
b/be/test/format_v2/table/iceberg_reader_test.cpp
index b66655832d7..7585ef73568 100644
--- a/be/test/format_v2/table/iceberg_reader_test.cpp
+++ b/be/test/format_v2/table/iceberg_reader_test.cpp
@@ -240,10 +240,12 @@ private:
class IcebergTableReaderMappingModeTestHelper final
: public doris::format::iceberg::IcebergTableReader {
public:
- TableColumnMappingMode
mapping_mode_for_schema(std::vector<ColumnDefinition> file_schema,
- TFileScanRangeParams*
scan_params = nullptr) {
+ TableColumnMappingMode mapping_mode_for_schema(
+ std::vector<ColumnDefinition> file_schema, TFileScanRangeParams*
scan_params = nullptr,
+ std::vector<ColumnDefinition> projected_columns = {}) {
_scan_params = scan_params;
_data_reader.file_schema = std::move(file_schema);
+ _projected_columns = std::move(projected_columns);
return mapping_mode();
}
};
@@ -818,6 +820,69 @@ void write_nested_equality_orc_file(const std::string&
file_path, const std::vec
output.write(memory_stream.getData(),
static_cast<std::streamsize>(memory_stream.getLength()));
}
+void write_nested_key_value_parquet_file(const std::string& file_path,
+ const std::vector<int32_t>& keys,
+ const std::vector<int32_t>& values) {
+ ASSERT_EQ(keys.size(), values.size());
+ auto key_field = arrow::field("delete_key", arrow::int32(), false)
+
->WithMetadata(arrow::key_value_metadata({"PARQUET:field_id"}, {"2"}));
+ auto value_field =
+ arrow::field("visible_value", arrow::int32(), false)
+
->WithMetadata(arrow::key_value_metadata({"PARQUET:field_id"}, {"3"}));
+ auto payload_result = arrow::StructArray::Make(
+ {build_int32_array(keys), build_int32_array(values)}, {key_field,
value_field});
+ ASSERT_TRUE(payload_result.ok()) << payload_result.status();
+ auto payload_field =
+ arrow::field("payload", arrow::struct_({key_field, value_field}),
false)
+
->WithMetadata(arrow::key_value_metadata({"PARQUET:field_id"}, {"1"}));
+ auto table = arrow::Table::Make(arrow::schema({payload_field}),
{*payload_result});
+
+ auto file_result = arrow::io::FileOutputStream::Open(file_path);
+ ASSERT_TRUE(file_result.ok()) << file_result.status();
+ std::shared_ptr<arrow::io::FileOutputStream> out = *file_result;
+ ::parquet::WriterProperties::Builder builder;
+ builder.version(::parquet::ParquetVersion::PARQUET_2_6);
+ builder.data_page_version(::parquet::ParquetDataPageVersion::V2);
+ builder.compression(::parquet::Compression::UNCOMPRESSED);
+ PARQUET_THROW_NOT_OK(::parquet::arrow::WriteTable(*table,
arrow::default_memory_pool(), out,
+
static_cast<int64_t>(keys.size()),
+ builder.build()));
+}
+
+void write_nested_key_value_orc_file(const std::string& file_path, const
std::vector<int64_t>& keys,
+ const std::vector<int64_t>& values) {
+ ASSERT_EQ(keys.size(), values.size());
+ auto type = std::unique_ptr<::orc::Type>(::orc::Type::buildTypeFromString(
+ "struct<payload:struct<delete_key:int,visible_value:int>>"));
+ type->getSubtype(0)->setAttribute("iceberg.id", "1");
+ type->getSubtype(0)->getSubtype(0)->setAttribute("iceberg.id", "2");
+ type->getSubtype(0)->getSubtype(1)->setAttribute("iceberg.id", "3");
+
+ MemoryOutputStream memory_stream(1024 * 1024);
+ ::orc::WriterOptions options;
+ options.setCompression(::orc::CompressionKind_NONE);
+ options.setMemoryPool(::orc::getDefaultPool());
+ auto writer = ::orc::createWriter(*type, &memory_stream, options);
+ auto batch = writer->createRowBatch(keys.size());
+ auto& root_batch = dynamic_cast<::orc::StructVectorBatch&>(*batch);
+ auto& payload_batch =
dynamic_cast<::orc::StructVectorBatch&>(*root_batch.fields[0]);
+ const std::array value_sets = {&keys, &values};
+ for (size_t field_idx = 0; field_idx < value_sets.size(); ++field_idx) {
+ auto& value_batch =
dynamic_cast<::orc::LongVectorBatch&>(*payload_batch.fields[field_idx]);
+ for (size_t row = 0; row < keys.size(); ++row) {
+ value_batch.data[row] = (*value_sets[field_idx])[row];
+ }
+ value_batch.numElements = keys.size();
+ }
+ root_batch.numElements = keys.size();
+ payload_batch.numElements = keys.size();
+ writer->add(*batch);
+ writer->close();
+
+ std::ofstream output(file_path, std::ios::binary);
+ output.write(memory_stream.getData(),
static_cast<std::streamsize>(memory_stream.getLength()));
+}
+
void write_timestamp_int_parquet_file(const std::string& file_path,
const std::vector<int64_t>& timestamps,
const std::vector<int32_t>& ids) {
@@ -1295,6 +1360,16 @@ void
init_iceberg_reader(doris::format::iceberg::IcebergTableReader* reader,
.ok());
}
+ColumnDefinition make_authoritatively_name_mapped_table_column(int32_t id,
std::string name,
+ const
DataTypePtr& type) {
+ auto column = make_table_column(id, name, type);
+ // Equality-delete tests need a readable result carrier while
independently exercising an
+ // ID-less hidden key. Make that carrier explicitly name-readable under
the V2 mapping rules.
+ column.name_mapping = {std::move(name)};
+ column.has_name_mapping = true;
+ return column;
+}
+
void expect_idless_equality_key_uses_delete_file_name(FileFormat file_format,
bool
authoritative_name_mapping) {
const std::string format_name = file_format == FileFormat::PARQUET ?
"parquet" : "orc";
@@ -1317,7 +1392,8 @@ void
expect_idless_equality_key_uses_delete_file_name(FileFormat file_format,
}
std::vector<ColumnDefinition> projected_columns;
- projected_columns.push_back(make_table_column(0, "id",
std::make_shared<DataTypeInt32>()));
+ projected_columns.push_back(make_authoritatively_name_mapped_table_column(
+ 0, "id", std::make_shared<DataTypeInt32>()));
auto equality_field = external_schema_field("future_name", 1, {}, "7");
if (authoritative_name_mapping) {
@@ -2798,6 +2874,25 @@ TEST(IcebergV2ReaderTest,
IcebergLegacyPlanKeepsAllFieldIdsMappingRule) {
TableColumnMappingMode::BY_NAME);
}
+TEST(IcebergV2ReaderTest, IcebergV2IdlessFileRequiresAuthoritativeNameMapping)
{
+ IcebergTableReaderMappingModeTestHelper reader;
+ TFileScanRangeParams scan_params;
+
scan_params.__set_iceberg_scan_semantics_version(ICEBERG_SCAN_SEMANTICS_VERSION_2);
+
+ auto file_column = make_file_column(1, "legacy_name",
std::make_shared<DataTypeInt32>());
+ file_column.identifier = Field {};
+ auto table_column = make_table_column(1, "current_name",
std::make_shared<DataTypeInt32>());
+
+ EXPECT_EQ(reader.mapping_mode_for_schema({file_column}, &scan_params,
{table_column}),
+ TableColumnMappingMode::BY_FIELD_ID);
+
+ table_column.has_name_mapping = true;
+ table_column.name_mapping = {"legacy_name"};
+ EXPECT_EQ(reader.mapping_mode_for_schema({std::move(file_column)},
&scan_params,
+ {std::move(table_column)}),
+ TableColumnMappingMode::BY_NAME);
+}
+
TEST(IcebergV2ReaderTest,
IcebergTableReaderDoesNotPushDownAggregateWithPositionDelete) {
const auto test_dir =
std::filesystem::temp_directory_path() /
"doris_iceberg_aggregate_position_delete_test";
@@ -3092,6 +3187,106 @@ TEST(IcebergV2ReaderTest,
IcebergNestedEqualityDeleteFiltersCurrentAndDroppedFie
run_case(FileFormat::PARQUET, false, false);
}
+// Keep the Parquet/ORC setup identical because both readers must preserve the
projected child
+// ordinal when an equality-delete key widens the same struct root after
mapper localization.
+//
NOLINTNEXTLINE(readability-function-cognitive-complexity,readability-function-size)
+TEST(IcebergV2ReaderTest,
IcebergNestedEqualityDeletePreservesVisibleSiblingProjection) {
+ const auto run_case = [](FileFormat file_format) {
+ const bool is_parquet = file_format == FileFormat::PARQUET;
+ const std::string format_name = is_parquet ? "parquet" : "orc";
+ const auto test_dir = std::filesystem::temp_directory_path() /
+ ("doris_v2_nested_equality_visible_sibling_" +
format_name);
+ std::filesystem::remove_all(test_dir);
+ std::filesystem::create_directories(test_dir);
+ const auto file_path = (test_dir / ("split." + format_name)).string();
+ const auto delete_file_path = (test_dir / ("equality-delete." +
format_name)).string();
+ if (is_parquet) {
+ write_nested_key_value_parquet_file(file_path, {100, 200, 300},
{10, 20, 30});
+ write_nested_equality_parquet_file(delete_file_path, {}, {200},
{false}, true,
+ "delete_key", 2);
+ } else {
+ write_nested_key_value_orc_file(file_path, {100, 200, 300}, {10,
20, 30});
+ write_nested_equality_orc_file(delete_file_path, {}, {200},
{false}, "delete_key", 2);
+ }
+
+ auto schema_payload = external_struct_schema_field(
+ "payload", 1,
+ {external_schema_field("delete_key", 2, {}, std::nullopt,
+
external_primitive_type(TPrimitiveType::INT)),
+ external_schema_field("visible_value", 3, {}, std::nullopt,
+
external_primitive_type(TPrimitiveType::INT))});
+ auto scan_params = make_local_scan_params(file_format);
+
scan_params.__set_iceberg_scan_semantics_version(ICEBERG_SCAN_SEMANTICS_VERSION_2);
+ scan_params.__set_current_schema_id(100);
+ scan_params.__set_history_schema_info({external_schema(100,
{schema_payload})});
+
+ const auto int_type = std::make_shared<DataTypeInt32>();
+ auto visible_value = make_table_column(3, "visible_value", int_type);
+ auto payload_type =
+ std::make_shared<DataTypeStruct>(DataTypes {int_type}, Strings
{"visible_value"});
+ auto payload = make_table_column(1, "payload", payload_type);
+ payload.children = {visible_value};
+ std::vector<ColumnDefinition> projected_columns = {payload};
+
+ RuntimeProfile profile("test_profile");
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ io::FileReaderStats file_reader_stats;
+ io::FileCacheStatistics file_cache_stats;
+ auto io_ctx = make_io_context(&file_reader_stats, &file_cache_stats);
+ ShardedKVCache cache(1);
+ doris::format::iceberg::IcebergTableReader reader;
+ init_iceberg_reader(&reader, projected_columns, &scan_params, io_ctx,
&state, &profile,
+ file_format);
+
+ auto split_options = build_split_options(file_path);
+ split_options.cache = &cache;
+ split_options.current_split_format = file_format;
+ const auto thrift_file_format =
+ is_parquet ? TFileFormatType::FORMAT_PARQUET :
TFileFormatType::FORMAT_ORC;
+
split_options.current_range.__set_table_format_params(make_iceberg_table_format_desc(
+ file_path,
+ {make_iceberg_equality_delete_file(delete_file_path, {2},
thrift_file_format)}, 3));
+ ASSERT_TRUE(reader.prepare_split(split_options).ok());
+
+ Block block = build_table_block(projected_columns);
+ bool eos = false;
+ ASSERT_TRUE(reader.get_block(&block, &eos).ok());
+ ASSERT_FALSE(eos);
+ const auto& payload_result =
+ assert_cast<const
ColumnStruct&>(expect_not_null_table_column(block, 0));
+ expect_int32_column_values(payload_result.get_column(0), {10, 30});
+
+ ASSERT_TRUE(reader.close().ok());
+ std::filesystem::remove_all(test_dir);
+ };
+
+ for (const auto file_format : {FileFormat::PARQUET, FileFormat::ORC}) {
+ run_case(file_format);
+ }
+}
+
+TEST(IcebergV2ReaderTest, IcebergMetadataOnlyMappingUsesHistoryNameMapping) {
+ IcebergTableReaderMappingModeTestHelper reader;
+ TFileScanRangeParams scan_params;
+
scan_params.__set_iceberg_scan_semantics_version(ICEBERG_SCAN_SEMANTICS_VERSION_2);
+
+ auto file_column = make_file_column(1, "legacy_key",
std::make_shared<DataTypeInt32>());
+ file_column.identifier = Field {};
+ EXPECT_EQ(reader.mapping_mode_for_schema({file_column}, &scan_params),
+ TableColumnMappingMode::BY_FIELD_ID);
+
+ auto key_field =
+ external_schema_field("current_key", 1, {"legacy_key"},
std::nullopt,
+
external_primitive_type(TPrimitiveType::INT), false, true);
+ key_field.field_ptr->__set_name_mapping_is_authoritative(true);
+ scan_params.__set_history_schema_info({external_schema(100,
{std::move(key_field)})});
+
+ // Metadata-only scans have no projected column carrying aliases, so the
complete schema must
+ // remain authoritative for resolving hidden equality-delete keys in
ID-less files.
+ EXPECT_EQ(reader.mapping_mode_for_schema({std::move(file_column)},
&scan_params),
+ TableColumnMappingMode::BY_NAME);
+}
+
// Keep the shared Parquet/ORC reader setup together so both V2 paths exercise
identical ID-less
// nested-name resolution.
//
NOLINTNEXTLINE(readability-function-cognitive-complexity,readability-function-size)
@@ -3138,7 +3333,8 @@ TEST(IcebergV2ReaderTest,
IcebergIdlessNestedEqualityKeyUsesAliasPathAndDeleteLe
}
std::vector<ColumnDefinition> projected_columns;
- projected_columns.push_back(make_table_column(0, "id",
std::make_shared<DataTypeInt32>()));
+
projected_columns.push_back(make_authoritatively_name_mapped_table_column(
+ 0, "id", std::make_shared<DataTypeInt32>()));
auto scan_params = make_local_scan_params(file_format);
scan_params.__set_iceberg_scan_semantics_version(ICEBERG_SCAN_SEMANTICS_VERSION_2);
scan_params.__set_current_schema_id(100);
@@ -3534,7 +3730,8 @@ TEST(IcebergV2ReaderTest,
IcebergEqualityDeleteMissingKeyDoesNotReadUnsupportedU
write_iceberg_equality_delete_parquet_file(delete_file_path, 1, 7,
"added_column");
std::vector<ColumnDefinition> projected_columns;
- projected_columns.push_back(make_table_column(0, "id",
std::make_shared<DataTypeInt32>()));
+ projected_columns.push_back(make_authoritatively_name_mapped_table_column(
+ 0, "id", std::make_shared<DataTypeInt32>()));
RuntimeProfile profile("test_profile");
RuntimeState state {TQueryOptions(), TQueryGlobals()};
@@ -4129,6 +4326,50 @@ TEST(IcebergV2ReaderTest,
ParquetReadsIdlessWrapperWithAuthoritativeEmptyMapping
std::filesystem::remove_all(test_dir);
}
+TEST(IcebergV2ReaderTest,
ParquetTreatsIdlessNestedFileAsMissingWithoutNameMapping) {
+ const auto test_dir = std::filesystem::temp_directory_path() /
+
"doris_iceberg_idless_nested_without_name_mapping_test";
+ std::filesystem::remove_all(test_dir);
+ std::filesystem::create_directories(test_dir);
+ const auto file_path = (test_dir / "split.parquet").string();
+ write_nested_equality_parquet_file(file_path, {}, {42}, {false}, true,
"leaf", 30, "outer",
+ false, false);
+
+ const auto int_type = std::make_shared<DataTypeInt32>();
+ auto leaf = make_table_column(30, "leaf", int_type);
+ auto outer_type = std::make_shared<DataTypeStruct>(DataTypes {int_type},
Strings {"leaf"});
+ auto outer = make_table_column(10, "outer", outer_type);
+ outer.children = {leaf};
+ std::vector<ColumnDefinition> projected_columns = {outer};
+
+ RuntimeProfile profile("test_profile");
+ RuntimeState state {TQueryOptions(), TQueryGlobals()};
+ auto scan_params = make_local_parquet_scan_params();
+
scan_params.__set_iceberg_scan_semantics_version(ICEBERG_SCAN_SEMANTICS_VERSION_2);
+ io::FileReaderStats file_reader_stats;
+ io::FileCacheStatistics file_cache_stats;
+ auto io_ctx = make_io_context(&file_reader_stats, &file_cache_stats);
+ ShardedKVCache cache(1);
+ doris::format::iceberg::IcebergTableReader reader;
+ init_iceberg_reader(&reader, projected_columns, &scan_params, io_ctx,
&state, &profile);
+ auto split_options = build_split_options(file_path);
+ split_options.cache = &cache;
+ split_options.current_range.__set_table_format_params(
+ make_iceberg_table_format_desc(file_path, {}));
+ ASSERT_TRUE(reader.prepare_split(split_options).ok());
+
+ Block block = build_table_block(projected_columns);
+ bool eos = false;
+ ASSERT_TRUE(reader.get_block(&block, &eos).ok());
+ ASSERT_FALSE(eos);
+ const auto& result = assert_cast<const
ColumnNullable&>(*block.get_by_position(0).column);
+ ASSERT_EQ(result.size(), 1);
+ EXPECT_EQ(result.get_null_map_data()[0], 1);
+
+ ASSERT_TRUE(reader.close().ok());
+ std::filesystem::remove_all(test_dir);
+}
+
TEST(IcebergV2ReaderTest,
ParquetUsesUnprojectedSiblingIdToRetainNullableWrapper) {
const auto test_dir = std::filesystem::temp_directory_path() /
"doris_iceberg_unprojected_sibling_id_wrapper_test";
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/iceberg_load/run01.sql
b/docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/iceberg_load/run01.sql
index bfe0a838131..4075719c954 100644
---
a/docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/iceberg_load/run01.sql
+++
b/docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/iceberg_load/run01.sql
@@ -9,6 +9,7 @@ drop table if exists multi_catalog.equality_delete_par_3;
drop table if exists multi_catalog.equality_delete_orc_1;
drop table if exists multi_catalog.equality_delete_orc_2;
drop table if exists multi_catalog.equality_delete_orc_3;
+drop table if exists multi_catalog.migrated_nested_without_name_mapping;
CALL system.register_table(
@@ -42,6 +43,13 @@ CALL system.register_table(
metadata_file =>
's3a://warehouse/wh/multi_catalog/equality_delete_orc_3/metadata/00010-f6ba4ee7-256f-41f3-8932-25ec703d8c8b.metadata.json'
);
+-- The imported Parquet file has no Iceberg field IDs, and the latest table
metadata intentionally
+-- omits schema.name-mapping.default to verify missing-field semantics after
migration.
+CALL system.register_table(
+ table => 'multi_catalog.migrated_nested_without_name_mapping',
+ metadata_file =>
's3a://warehouse/wh/multi_catalog/migrated_nested_without_name_mapping/metadata/00003-97f3e871-e421-48ed-a735-354f4e1c9502.metadata.json'
+);
+
-- flink
-- CREATE CATALOG iceberg_rest
-- WITH (
@@ -199,4 +207,4 @@ CALL system.register_table(
-- ALTER TABLE equality_delete_orc_3 MODIFY (
-- `new_id` INT , `name` string ,`data` STRING ,
-- PRIMARY KEY (new_id, `data`) NOT ENFORCED);
--- insert into equality_delete_orc_3 values(1, 'smith4', 'aaa');
\ No newline at end of file
+-- insert into equality_delete_orc_3 values(1, 'smith4', 'aaa');
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/iceberg/migrated_nested_without_name_mapping/data/data.parquet
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/iceberg/migrated_nested_without_name_mapping/data/data.parquet
new file mode 100644
index 00000000000..0311c6f2595
Binary files /dev/null and
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/iceberg/migrated_nested_without_name_mapping/data/data.parquet
differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/iceberg/migrated_nested_without_name_mapping/metadata/00000-4c8aafe0-9de1-4b3f-942d-305dc33fb82f.metadata.json
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/iceberg/migrated_nested_without_name_mapping/metadata/00000-4c8aafe0-9de1-4b3f-942d-305dc33fb82f.metadata.json
new file mode 100644
index 00000000000..57b95fe8f77
--- /dev/null
+++
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/iceberg/migrated_nested_without_name_mapping/metadata/00000-4c8aafe0-9de1-4b3f-942d-305dc33fb82f.metadata.json
@@ -0,0 +1 @@
+{"format-version":2,"table-uuid":"bd7ffeda-37f2-4199-92e0-1b97e7ddf643","location":"s3a://warehouse/wh/multi_catalog/migrated_nested_without_name_mapping","last-sequence-number":0,"last-updated-ms":1788760308700,"last-column-id":4,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"int"},{"id":2,"name":"nested_struct","required":false,"type":{"type":"struct","fields":[{"id":3,"name":"value_a","required":false,"type":"int"
[...]
\ No newline at end of file
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/iceberg/migrated_nested_without_name_mapping/metadata/00001-94d09fba-61c6-4e4b-b0f5-6023f70d5033.metadata.json
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/iceberg/migrated_nested_without_name_mapping/metadata/00001-94d09fba-61c6-4e4b-b0f5-6023f70d5033.metadata.json
new file mode 100644
index 00000000000..9f50c4f5d4e
--- /dev/null
+++
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/iceberg/migrated_nested_without_name_mapping/metadata/00001-94d09fba-61c6-4e4b-b0f5-6023f70d5033.metadata.json
@@ -0,0 +1 @@
+{"format-version":2,"table-uuid":"bd7ffeda-37f2-4199-92e0-1b97e7ddf643","location":"s3a://warehouse/wh/multi_catalog/migrated_nested_without_name_mapping","last-sequence-number":0,"last-updated-ms":1788760308887,"last-column-id":4,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"int"},{"id":2,"name":"nested_struct","required":false,"type":{"type":"struct","fields":[{"id":3,"name":"value_a","required":false,"type":"int"
[...]
\ No newline at end of file
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/iceberg/migrated_nested_without_name_mapping/metadata/00002-7493fffe-54f9-486a-b15e-7333fdb96f61.metadata.json
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/iceberg/migrated_nested_without_name_mapping/metadata/00002-7493fffe-54f9-486a-b15e-7333fdb96f61.metadata.json
new file mode 100644
index 00000000000..b3fca919fff
--- /dev/null
+++
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/iceberg/migrated_nested_without_name_mapping/metadata/00002-7493fffe-54f9-486a-b15e-7333fdb96f61.metadata.json
@@ -0,0 +1 @@
+{"format-version":2,"table-uuid":"bd7ffeda-37f2-4199-92e0-1b97e7ddf643","location":"s3a://warehouse/wh/multi_catalog/migrated_nested_without_name_mapping","last-sequence-number":1,"last-updated-ms":1788760313465,"last-column-id":4,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"int"},{"id":2,"name":"nested_struct","required":false,"type":{"type":"struct","fields":[{"id":3,"name":"value_a","required":false,"type":"int"
[...]
\ No newline at end of file
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/iceberg/migrated_nested_without_name_mapping/metadata/00003-97f3e871-e421-48ed-a735-354f4e1c9502.metadata.json
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/iceberg/migrated_nested_without_name_mapping/metadata/00003-97f3e871-e421-48ed-a735-354f4e1c9502.metadata.json
new file mode 100644
index 00000000000..5c706e40195
--- /dev/null
+++
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/iceberg/migrated_nested_without_name_mapping/metadata/00003-97f3e871-e421-48ed-a735-354f4e1c9502.metadata.json
@@ -0,0 +1 @@
+{"format-version":2,"table-uuid":"bd7ffeda-37f2-4199-92e0-1b97e7ddf643","location":"s3a://warehouse/wh/multi_catalog/migrated_nested_without_name_mapping","last-sequence-number":1,"last-updated-ms":1788760313716,"last-column-id":4,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"int"},{"id":2,"name":"nested_struct","required":false,"type":{"type":"struct","fields":[{"id":3,"name":"value_a","required":false,"type":"int"
[...]
\ No newline at end of file
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/iceberg/migrated_nested_without_name_mapping/metadata/snap-2748466134835934544-1-94f693c5-6efb-4e9d-a40e-f02ac301fc14.avro
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/iceberg/migrated_nested_without_name_mapping/metadata/snap-2748466134835934544-1-94f693c5-6efb-4e9d-a40e-f02ac301fc14.avro
new file mode 100644
index 00000000000..b24d91edf04
Binary files /dev/null and
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/iceberg/migrated_nested_without_name_mapping/metadata/snap-2748466134835934544-1-94f693c5-6efb-4e9d-a40e-f02ac301fc14.avro
differ
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/iceberg/migrated_nested_without_name_mapping/metadata/stage-7-task-401-manifest-d04009d1-83e2-4d2a-859e-79d57e8dd381.avro
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/iceberg/migrated_nested_without_name_mapping/metadata/stage-7-task-401-manifest-d04009d1-83e2-4d2a-859e-79d57e8dd381.avro
new file mode 100644
index 00000000000..d087d3f68c0
Binary files /dev/null and
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/iceberg/migrated_nested_without_name_mapping/metadata/stage-7-task-401-manifest-d04009d1-83e2-4d2a-859e-79d57e8dd381.avro
differ
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java
index a43ce9dc9d2..da4513ff8a1 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java
@@ -117,6 +117,7 @@ import org.apache.iceberg.hive.HiveCatalog;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.mapping.MappedField;
import org.apache.iceberg.mapping.MappedFields;
+import org.apache.iceberg.mapping.MappingUtil;
import org.apache.iceberg.mapping.NameMapping;
import org.apache.iceberg.mapping.NameMappingParser;
import org.apache.iceberg.transforms.Transforms;
@@ -2352,8 +2353,13 @@ public class IcebergUtils {
extractMappingsFromNameMapping(mapping.asMappedFields(), result);
return Optional.of(result);
} catch (Exception e) {
+ // Keep ID-less files readable by current names when a malformed
property cannot provide
+ // authoritative aliases; Optional.empty() must remain reserved
for an absent property.
LOG.warn("Failed to parse name mapping from Iceberg table
properties", e);
- return Optional.empty();
+ Map<Integer, List<String>> fallback = new HashMap<>();
+ extractMappingsFromNameMapping(
+
MappingUtil.create(icebergTable.schema()).asMappedFields(), fallback);
+ return Optional.of(fallback);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
index a4fe894e927..fd719d84b4d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
@@ -3835,6 +3835,9 @@ public class SessionVariable implements Serializable,
Writable {
this.useSerialExchange = random.nextBoolean();
this.enableCommonExpPushDownForInvertedIndex = random.nextBoolean();
this.enableExprZonemapFilter = Config.pull_request_id % 2 == 0;
+ // Fuzzy sessions must exercise the production-default V2 path
consistently. Dedicated
+ // compatibility cases can still select the legacy scanner explicitly
after initialization.
+ this.enableFileScannerV2 = true;
this.disableStreamPreaggregations = random.nextBoolean();
this.enableStreamingAggHashJoinForcePassthrough = random.nextBoolean();
this.enableLocalExchangeBeforeAgg = random.nextBoolean();
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergUtilsTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergUtilsTest.java
index 393d60cf6f5..616f1eefa5f 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergUtilsTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergUtilsTest.java
@@ -173,6 +173,23 @@ public class IcebergUtilsTest {
Mockito.verify(operations, Mockito.times(1)).current();
}
+ @Test
+ public void testMalformedNameMappingFallsBackToCurrentSchemaNames() {
+ Schema schema = new Schema(
+ Types.NestedField.required(1, "id", Types.IntegerType.get()),
+ Types.NestedField.optional(2, "name", Types.StringType.get()));
+ Table table = Mockito.mock(Table.class);
+ Mockito.when(table.properties()).thenReturn(Collections.singletonMap(
+ TableProperties.DEFAULT_NAME_MAPPING, "{not valid json"));
+ Mockito.when(table.schema()).thenReturn(schema);
+
+ Optional<Map<Integer, List<String>>> mapping =
IcebergUtils.getNameMapping(table);
+ Assert.assertTrue(mapping.isPresent());
+ Map<Integer, List<String>> fallback = mapping.get();
+ Assert.assertEquals(Collections.singletonList("id"), fallback.get(1));
+ Assert.assertEquals(Collections.singletonList("name"),
fallback.get(2));
+ }
+
@Test
public void testGetFileFormatUsesPropertiesWithoutPlanningDataFiles() {
Table table = Mockito.mock(Table.class);
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/qe/SessionVariablesTest.java
b/fe/fe-core/src/test/java/org/apache/doris/qe/SessionVariablesTest.java
index 52e7b4c1a43..1e47c3a4c72 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/qe/SessionVariablesTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/qe/SessionVariablesTest.java
@@ -204,6 +204,7 @@ public class SessionVariablesTest extends TestWithFeService
{
VariableMgr.VarAttr varAttr =
field.getAnnotation(VariableMgr.VarAttr.class);
Assertions.assertFalse(varAttr.fuzzy());
+ sessionVar.enableFileScannerV2 = false;
sessionVar.initFuzzyModeVariables();
Assertions.assertTrue(sessionVar.enableFileScannerV2);
}
diff --git
a/regression-test/data/external_table_p0/iceberg/test_gen_iceberg_by_api.out
b/regression-test/data/external_table_p0/iceberg/test_gen_iceberg_by_api.out
index b3d42dfcd19..21218a4e51c 100644
--- a/regression-test/data/external_table_p0/iceberg/test_gen_iceberg_by_api.out
+++ b/regression-test/data/external_table_p0/iceberg/test_gen_iceberg_by_api.out
@@ -5,6 +5,3 @@
2 1970-01-03T09:02:04.000001 c
2 1970-01-03T09:02:04.000001 d
--- !q02 --
-463870
-
diff --git
a/regression-test/data/external_table_p0/iceberg/test_iceberg_migrated_nested_without_name_mapping.out
b/regression-test/data/external_table_p0/iceberg/test_iceberg_migrated_nested_without_name_mapping.out
new file mode 100644
index 00000000000..9aea1c252d0
--- /dev/null
+++
b/regression-test/data/external_table_p0/iceberg/test_iceberg_migrated_nested_without_name_mapping.out
@@ -0,0 +1,3 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !idless_nested_fields_are_missing --
+\N \N \N
diff --git
a/regression-test/data/external_table_p0/iceberg/test_iceberg_struct_schema_evolution.out
b/regression-test/data/external_table_p0/iceberg/test_iceberg_struct_schema_evolution.out
index a364316df42..b106ac908a9 100644
---
a/regression-test/data/external_table_p0/iceberg/test_iceberg_struct_schema_evolution.out
+++
b/regression-test/data/external_table_p0/iceberg/test_iceberg_struct_schema_evolution.out
@@ -39,6 +39,9 @@ a_struct
struct<renamed:bigint,keep:bigint,drop_and_add:bigint,added:bigint> Yes
-- !struct_predicate_4 --
2
+-- !struct_projected_siblings_with_missing_predicate --
+11 12
+
-- !struct_multi --
11 12 \N \N
21 22 23 24
diff --git
a/regression-test/suites/external_table_p0/iceberg/test_gen_iceberg_by_api.groovy
b/regression-test/suites/external_table_p0/iceberg/test_gen_iceberg_by_api.groovy
index 512adecf7f3..4f0bf7c1ca2 100644
---
a/regression-test/suites/external_table_p0/iceberg/test_gen_iceberg_by_api.groovy
+++
b/regression-test/suites/external_table_p0/iceberg/test_gen_iceberg_by_api.groovy
@@ -45,10 +45,11 @@ suite("test_gen_iceberg_by_api",
"p0,external,doris,external_docker,external_doc
def q01 = {
qt_q01 """ select * from multi_partition2 order by val """
- try {
- qt_q02 """ select count(*) from table_with_append_file where
MAN_ID is not null """
- } catch (Exception e) {
- assertTrue(e.getMessage().contains("name_mapping must be set when
read missing field id data file."), e.getMessage());
+ test {
+ sql """ select count(*) from table_with_append_file where MAN_ID
is not null """
+ // This fixture has no field IDs or authoritative name mapping, so
its required
+ // columns must be treated as missing instead of being matched by
their current names.
+ exception "Missing required field: MAN_ID"
}
}
diff --git
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_migrated_nested_without_name_mapping.groovy
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_migrated_nested_without_name_mapping.groovy
new file mode 100644
index 00000000000..8f1ceec0906
--- /dev/null
+++
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_migrated_nested_without_name_mapping.groovy
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_iceberg_migrated_nested_without_name_mapping", "p0,external") {
+ String enabled = context.config.otherConfigs.get("enableIcebergTest")
+ if (enabled == null || !enabled.equalsIgnoreCase("true")) {
+ logger.info("disable iceberg test.")
+ return
+ }
+
+ String rest_port = context.config.otherConfigs.get("iceberg_rest_uri_port")
+ String minio_port = context.config.otherConfigs.get("iceberg_minio_port")
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+ String catalog_name = "test_iceberg_migrated_nested_without_name_mapping"
+
+ sql """drop catalog if exists ${catalog_name}"""
+ sql """
+ CREATE CATALOG ${catalog_name} PROPERTIES (
+ 'type'='iceberg',
+ 'iceberg.catalog.type'='rest',
+ 'uri' = 'http://${externalEnvIp}:${rest_port}',
+ 's3.access_key' = 'admin',
+ 's3.secret_key' = 'password',
+ 's3.endpoint' = 'http://${externalEnvIp}:${minio_port}',
+ 's3.region' = 'us-east-1'
+ )
+ """
+ sql """switch ${catalog_name}"""
+ sql """use multi_catalog"""
+ sql """set enable_fallback_to_original_planner=false"""
+
+ // The physical Parquet fields predate migration and have no Iceberg IDs.
Once the explicit
+ // default name mapping is removed, neither the root nor nested values may
match by current name.
+ qt_idless_nested_fields_are_missing """
+ SELECT id, nested_struct.value_a, nested_struct.value_b
+ FROM migrated_nested_without_name_mapping
+ ORDER BY id
+ """
+
+ sql """drop catalog if exists ${catalog_name}"""
+}
diff --git
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_struct_schema_evolution.groovy
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_struct_schema_evolution.groovy
index e6142771df4..f9dab204421 100644
---
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_struct_schema_evolution.groovy
+++
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_struct_schema_evolution.groovy
@@ -89,6 +89,15 @@ suite("test_iceberg_struct_schema_evolution",
"p0,external,doris,external_docker
qt_struct_predicate_3 """SELECT id FROM ${table_name} WHERE
element_at(a_struct, 'added') IS NULL ORDER BY id"""
qt_struct_predicate_4 """SELECT id FROM ${table_name} WHERE
element_at(a_struct, 'added') IS NOT NULL ORDER BY id"""
+ // A missing predicate child widens the physical scan back to the full
struct. The projected
+ // sibling fields must keep their Iceberg field-id mapping instead of
shifting by file ordinal.
+ qt_struct_projected_siblings_with_missing_predicate """
+ SELECT element_at(a_struct, 'renamed'), element_at(a_struct, 'keep')
+ FROM ${table_name}
+ WHERE element_at(a_struct, 'added') IS NULL
+ ORDER BY id
+ """
+
// Test 7: Multiple struct fields in one query
qt_struct_multi """SELECT element_at(a_struct, 'renamed'),
element_at(a_struct, 'keep'), element_at(a_struct, 'drop_and_add'),
element_at(a_struct, 'added') FROM ${table_name} ORDER BY id"""
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
