Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/8070#discussion_r36700528
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/parquet/CatalystRowConverter.scala
---
@@ -446,4 +478,61 @@ private[parquet] class CatalystRowConverter(
}
}
}
+
+ private trait RepeatedConverter {
+ private var currentArray: ArrayBuffer[Any] = _
+
+ protected def newArrayUpdater(updater: ParentContainerUpdater) = new
ParentContainerUpdater {
+ override def start(): Unit = currentArray = ArrayBuffer.empty[Any]
+ override def end(): Unit = updater.set(new
GenericArrayData(currentArray.toArray))
+ override def set(value: Any): Unit = currentArray += value
+ }
+ }
+
+ /**
+ * A primitive converter for converting unannotated repeated primitive
values to required arrays
+ * of required primitives values.
+ */
+ private final class RepeatedPrimitiveConverter(
+ parquetType: Type,
+ catalystType: DataType,
+ parentUpdater: ParentContainerUpdater)
+ extends PrimitiveConverter with RepeatedConverter with
HasParentContainerUpdater {
--- End diff --
It's because `CatalystPrimitiveConverter` is defined as:
```scala
class CatalystPrimitiveConverter(val updater: ParentContainerUpdater)
extends
private[parquet] class CatalystPrimitiveConverter(val updater:
ParentContainerUpdater)
extends PrimitiveConverter with HasParentContainerUpdater {
...
}
```
the `val updater` part has two meanings:
1. `updater` is made a constructor argument, and
2. `def updater` in `HasParentContainerUpdater` is overriden since
`updater` is a read-only `val`.
The 2nd fact prevents subclasses of `CatalystPrimitiveConverter` to
override the `updater` field.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
