RussellSpitzer commented on a change in pull request #1421:
URL: https://github.com/apache/iceberg/pull/1421#discussion_r511128848
##########
File path: spark/src/main/java/org/apache/iceberg/spark/SparkDataFile.java
##########
@@ -46,18 +47,38 @@
private final int upperBoundsPosition;
private final int keyMetadataPosition;
private final int splitOffsetsPosition;
+ private final int specIdPosition;
private final Type lowerBoundsType;
private final Type upperBoundsType;
private final Type keyMetadataType;
private final SparkStructLike wrappedPartition;
+ private final Types.StructType partitionStruct;
private Row wrapped;
+ private static final StructLike EMPTY_PARTITION_INFO = new StructLike() {
+ @Override
+ public int size() {
+ return 0;
+ }
+
+ @Override
+ public <T> T get(int pos, Class<T> javaClass) {
+ throw new UnsupportedOperationException("Cannot get a value from an
empty partition");
+ }
+
+ @Override
+ public <T> void set(int pos, T value) {
+ throw new UnsupportedOperationException("Cannot set a value in an empty
partition");
+ }
+ };
+
public SparkDataFile(Types.StructType type, StructType sparkType) {
this.lowerBoundsType = type.fieldType("lower_bounds");
this.upperBoundsType = type.fieldType("upper_bounds");
this.keyMetadataType = type.fieldType("key_metadata");
- this.wrappedPartition = new
SparkStructLike(type.fieldType("partition").asStructType());
+ this.partitionStruct = type.fieldType("partition").asStructType();
Review comment:
This is necessary for the "Copy" method which is needed for building the
BaseScanTasks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
