c21 commented on a change in pull request #35002:
URL: https://github.com/apache/spark/pull/35002#discussion_r775301600
##########
File path:
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcMapColumnVector.java
##########
@@ -32,28 +33,30 @@
public class OrcMapColumnVector extends OrcColumnVector {
private final OrcColumnVector keys;
private final OrcColumnVector values;
- private final long[] offsets;
- private final long[] lengths;
+ private MapColumnVector mapData;
OrcMapColumnVector(
DataType type,
ColumnVector vector,
OrcColumnVector keys,
- OrcColumnVector values,
- long[] offsets,
- long[] lengths) {
+ OrcColumnVector values) {
super(type, vector);
this.keys = keys;
this.values = values;
- this.offsets = offsets;
- this.lengths = lengths;
+
+ if (vector instanceof MapColumnVector) {
+ mapData = (MapColumnVector) vector;
+ } else {
+ throw new UnsupportedOperationException();
+ }
}
@Override
public ColumnarMap getMap(int ordinal) {
- return new ColumnarMap(keys, values, (int) offsets[ordinal], (int)
lengths[ordinal]);
+ return new ColumnarMap(keys, values, (int) mapData.offsets[ordinal],
Review comment:
```java
public ColumnarMap getMap(int ordinal) {
int offset = (int) ((MapColumnVector) baseData).offsets[ordinal];
int length = (int) ((MapColumnVector) baseData).lengths[ordinal];
return new ColumnarMap(keys, values, offset, length);
}
```
##########
File path:
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcArrayColumnVector.java
##########
@@ -31,26 +32,27 @@
*/
public class OrcArrayColumnVector extends OrcColumnVector {
private final OrcColumnVector data;
- private final long[] offsets;
- private final long[] lengths;
+ private ListColumnVector listData;
OrcArrayColumnVector(
Review comment:
I think we don't need to store another copy of `vector` here. We can
change
[`OrcColumnVector.baseData`](https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnVector.java#L32)
from `private` to `protected`, and just use `baseData` to get offset and
length. So how about defining the constructor like below:
```java
OrcArrayColumnVector(
DataType type,
ListColumnVector vector,
OrcColumnVector data) {
super(type, vector);
this.data = data;
}
```
##########
File path:
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcMapColumnVector.java
##########
@@ -32,28 +33,30 @@
public class OrcMapColumnVector extends OrcColumnVector {
private final OrcColumnVector keys;
private final OrcColumnVector values;
- private final long[] offsets;
- private final long[] lengths;
+ private MapColumnVector mapData;
OrcMapColumnVector(
Review comment:
Similarly we can define `OrcMapColumnVector` constructor as below:
```java
OrcMapColumnVector(
DataType type,
MapColumnVector vector,
OrcColumnVector keys,
OrcColumnVector values) {
super(type, vector);
this.keys = keys;
this.values = values;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
