rdblue commented on a change in pull request #1955:
URL: https://github.com/apache/iceberg/pull/1955#discussion_r546158023
##########
File path:
spark/src/main/java/org/apache/iceberg/spark/source/BatchDataReader.java
##########
@@ -68,18 +68,7 @@
// update the current file for Spark's filename() function
InputFileBlockHolder.set(file.path().toString(), task.start(),
task.length());
- // schema or rows returned by readers
- PartitionSpec spec = task.spec();
- Set<Integer> idColumns = spec.identitySourceIds();
- Schema partitionSchema = TypeUtil.select(expectedSchema, idColumns);
- boolean projectsIdentityPartitionColumns =
!partitionSchema.columns().isEmpty();
-
- Map<Integer, ?> idToConstant;
- if (projectsIdentityPartitionColumns) {
- idToConstant = PartitionUtil.constantsMap(task,
BatchDataReader::convertConstant);
- } else {
- idToConstant = ImmutableMap.of();
- }
+ Map<Integer, ?> idToConstant = PartitionUtil.constantsMap(task,
BatchDataReader::convertConstant);
Review comment:
It isn't necessary to check whether there are projected ID columns. The
code is shorter if the values are available by default, even if they aren't
used. This fixes the problem where there are constants to add (like `_file`)
but no identity partition values are projected.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
