aokolnychyi commented on code in PR #15297:
URL: https://github.com/apache/iceberg/pull/15297#discussion_r2794458650
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java:
##########
@@ -320,74 +319,63 @@ private boolean
metricsModeSupportsAggregatePushDown(List<BoundAggregate<?, ?>>
@Override
public void pruneColumns(StructType requestedSchema) {
- StructType requestedProjection =
- new StructType(
- Stream.of(requestedSchema.fields())
- .filter(field ->
MetadataColumns.nonMetadataColumn(field.name()))
- .toArray(StructField[]::new));
-
- // the projection should include all columns that will be returned,
including those only used in
- // filters
- this.schema =
- SparkSchemaUtil.prune(schema, requestedProjection, filterExpression(),
caseSensitive);
-
- Stream.of(requestedSchema.fields())
- .map(StructField::name)
- .filter(MetadataColumns::isMetadataColumn)
- .distinct()
- .forEach(metaColumns::add);
- }
-
- private Schema schemaWithMetadataColumns() {
- // metadata columns
- List<Types.NestedField> metadataFields =
- metaColumns.stream()
- .distinct()
- .map(name -> MetadataColumns.metadataColumn(table, name))
- .collect(Collectors.toList());
- Schema metadataSchema = calculateMetadataSchema(metadataFields);
-
- // schema or rows returned by readers
- return TypeUtil.join(schema, metadataSchema);
- }
-
- private Schema calculateMetadataSchema(List<Types.NestedField>
metaColumnFields) {
- Optional<Types.NestedField> partitionField =
Review Comment:
A lot of this logic is not specific to Spark and it was a bit harder to
navigate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
