rdblue commented on a change in pull request #1335:
URL: https://github.com/apache/iceberg/pull/1335#discussion_r470193713
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetUtil.java
##########
@@ -136,6 +154,19 @@ public static Metrics footerMetrics(ParquetMetadata
metadata, MetricsConfig metr
toBufferMap(fileSchema, lowerBounds), toBufferMap(fileSchema,
upperBounds));
}
+ private static MessageType getParquetType(ParquetMetadata metadata, Schema
expectedSchema, NameMapping nameMapping) {
+ MessageType type = metadata.getFileMetaData().getSchema();
+ if (ParquetSchemaUtil.hasIds(type)) {
+ return expectedSchema != null ? ParquetSchemaUtil.pruneColumns(type,
expectedSchema) : type;
+ } else if (nameMapping != null) {
+ MessageType typeWithIds = ParquetSchemaUtil.applyNameMapping(type,
nameMapping);
Review comment:
Do we need `expectedSchema` to do this?
It seems like it would be easier if we didn't need it. What we want is to
use the IDs from the mapping without assigning new IDs for columns that don't
have them during conversion. So why not make a version of `convert` that
doesn't assign columns? That seems much easier than doing column pruning and it
doesn't require a new schema -- which introduced new methods everywhere to pass
that new schema.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
