github-actions[bot] commented on code in PR #65262:
URL: https://github.com/apache/doris/pull/65262#discussion_r3570217958
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java:
##########
@@ -596,11 +602,68 @@ public TableScan createTableScan() throws UserException {
this.pushdownIcebergPredicates.add(predicate.toString());
}
+ // Doris reads normal Iceberg table files in BE and applies column
pruning through scan range params.
+ // System tables are different: Iceberg SDK DataTask materializes rows
using the projected scan
+ // schema. Keep Doris required columns as the projection prefix so the
JNI reader can read rows by
+ // required ordinal, and append filter-only columns after them for
Iceberg residual evaluation.
+ if (isSystemTable) {
+ Schema projectedSchema =
getSystemTableProjectedSchema(expressions, scan.isCaseSensitive());
+ if (!projectedSchema.columns().isEmpty()) {
+ scan = scan.project(projectedSchema);
+ }
+ }
+
icebergTableScan =
scan.planWith(source.getCatalog().getThreadPoolWithPreAuth());
return icebergTableScan;
}
+ private Schema getSystemTableProjectedSchema(List<Expression> expressions,
boolean caseSensitive)
+ throws UserException {
+ List<NestedField> projectedFields = new ArrayList<>();
+ Set<Integer> projectedFieldIds = new HashSet<>();
+ for (SlotDescriptor slot : desc.getSlots()) {
+ Column column = slot.getColumn();
+ String columnName = column.getName();
+ if (Column.ICEBERG_ROWID_COL.equalsIgnoreCase(columnName)
+ || columnName.startsWith(Column.GLOBAL_ROWID_COL)
+ || IcebergUtils.isIcebergRowLineageColumn(column)) {
+ continue;
+ }
+
+ NestedField field = icebergTable.schema().findField(columnName);
+ if (field == null) {
+ throw new UserException("Column " + columnName + " not found
in Iceberg system table schema");
+ }
+ if (projectedFieldIds.add(field.fieldId())) {
+ projectedFields.add(field);
+ }
Review Comment:
Here we still add the full Iceberg top-level field to the SDK projection
even when Doris has pruned the scan slot to a narrower nested type. System
tables opt into nested-column pruning, and the translator stores that pruned
type in the `SlotDescriptor`; `ExternalUtil` then serializes only those pruned
struct children, so the JNI vector layer builds struct indexes as
`0..children.size()-1`. But the Java scanner receives the full Iceberg
`StructLike`, and `IcebergSysTableColumnValue.unpackStruct()` applies those
pruned indexes against the full struct. A query that materializes only a
non-first nested metadata child, such as the second field inside a multi-column
`$files.partition` struct, will read the wrong child value. Please either
project the nested Iceberg field with the same pruned child IDs, or disable
nested pruning for `IcebergSysExternalTable` until the JNI path can map pruned
children back to their original Iceberg positions. The regression should also
cover a non-first
nested metadata child.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
