Fokko commented on code in PR #7553:
URL: https://github.com/apache/iceberg/pull/7553#discussion_r1199218126
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/PruneColumnsWithoutReordering.java:
##########
@@ -24,26 +24,30 @@
import org.apache.iceberg.Schema;
import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
import org.apache.iceberg.types.Type;
import org.apache.iceberg.types.Type.TypeID;
import org.apache.iceberg.types.TypeUtil;
import org.apache.iceberg.types.Types;
+import org.apache.spark.sql.types.AbstractDataType;
import org.apache.spark.sql.types.ArrayType;
-import org.apache.spark.sql.types.BinaryType;
-import org.apache.spark.sql.types.BooleanType;
+import org.apache.spark.sql.types.BinaryType$;
Review Comment:
Before we did a `.instanceOf` which also checks the subclass. Since the
TypeID of the `TIMESTAMP` can be both:
```java
private static final ImmutableMap<TypeID, Set<Class<? extends
AbstractDataType>>> TYPES =
ImmutableMap.<TypeID, Set<Class<? extends AbstractDataType>>>builder()
...
.put(TypeID.TIMESTAMP, ImmutableSet.of(TimestampType$.class,
TimestampNTZType$.class))
```
We need to check for the type.
Since iterating over the set is wasteful, I check if the class is in the
Set, but then it won't check for the subtypes:
```java
// Only checks the actual type:
expectedType.contains(current.getClass())
```
It actually returns the Scala one:
```
@Stable
case object BinaryType$ extends BinaryType
```
Therefore this needed to change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
