stevenzwu commented on code in PR #13445:
URL: https://github.com/apache/iceberg/pull/13445#discussion_r2299047920
##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/data/SparkParquetWriters.java:
##########
@@ -613,13 +617,32 @@ private static class InternalRowWriter extends
ParquetValueWriters.StructWriter<
private final DataType[] types;
private InternalRowWriter(List<ParquetValueWriter<?>> writers,
List<DataType> types) {
- super(writers);
+ super(writerToFieldIndex(types, writers.size()), writers);
this.types = types.toArray(new DataType[0]);
}
@Override
protected Object get(InternalRow struct, int index) {
return struct.get(index, types[index]);
}
+
+ /** Returns a mapping from writer index to field index, skipping Unknown
columns. */
+ private static int[] writerToFieldIndex(List<DataType> types, int
numWriters) {
+ if (null == types) {
+ return IntStream.rangeClosed(0, numWriters).toArray();
Review Comment:
should it be `range` or `rangeClosed`? I though the end should be exclusive
as `[0, numWriters)` .
##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/data/SparkParquetWriters.java:
##########
@@ -613,13 +617,32 @@ private static class InternalRowWriter extends
ParquetValueWriters.StructWriter<
private final DataType[] types;
private InternalRowWriter(List<ParquetValueWriter<?>> writers,
List<DataType> types) {
- super(writers);
+ super(writerToFieldIndex(types, writers.size()), writers);
this.types = types.toArray(new DataType[0]);
Review Comment:
nit: not directly caused by this PR. since we are converting types to an
array, maybe we should just change the construct org to `DataType[] types`.
Caller can collect the stream using `.toArray(DataType[]::new)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
