Github user dbtsai commented on a diff in the pull request:
https://github.com/apache/spark/pull/21952#discussion_r207444037
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -100,17 +100,20 @@ class AvroSerializer(rootCatalystType: DataType,
rootAvroType: Schema, nullable:
et, resolveNullableType(avroType.getElementType, containsNull))
(getter, ordinal) => {
val arrayData = getter.getArray(ordinal)
- val result = new java.util.ArrayList[Any]
--- End diff --
My previous experience in ml project told me that `ArrayList` has slower
setter performance due to one extra function call, so my preference is using
array as much as possible, and wrap it into the right container in the end.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
