rdsr commented on a change in pull request #843: [WIP] InputFormat support for
Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/843#discussion_r402418755
##########
File path:
mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java
##########
@@ -346,14 +348,15 @@ public void close() throws IOException {
DataFile file = currentTask.file();
// schema of rows returned by readers
PartitionSpec spec = currentTask.spec();
- Set<Integer> idColumns = spec.identitySourceIds();
Schema readSchema = expectedSchema != null ? expectedSchema :
tableSchema;
+ Set<Integer> idColumns = Sets.intersection(spec.identitySourceIds(),
TypeUtil.getProjectedIds(readSchema));
boolean hasJoinedPartitionColumns = !idColumns.isEmpty();
+
if (hasJoinedPartitionColumns) {
- readSchema = TypeUtil.selectNot(tableSchema, idColumns);
Review comment:
Had to change this logic slightly so that whatever schema the user
projected, that is what is returned [we drop identity partition columns not
projected by the user]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
