Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/13758
Thank you for your comment. We are on the same page to address the problem.
In this case, you are right. This is because we finally would like to write
an primitive int array `mapelements_value` into `UnsafeArrayData` at
projection. If I am correct, current projection assumes to receive `ArrayData`
from the preview phase and to handle the `ArrayData` by using
`UnsafeArrayWriter`. They can take care of `nullability`.
I like to directly create `UnsafeArrayData` from an primitive array? How
can we implement this? Can we pass a primitive array to Projection? Or, can we
pass `UnsafeArrayData`, which has been created from a primitive array, to
Projection? What do you think?
FYI: here is a pair of logical and physical plans.
```
== Analyzed Logical Plan ==
value: array<int>
SerializeFromObject [newInstance(class
org.apache.spark.sql.catalyst.util.GenericArrayData) AS value#7]
+- MapElements <function1>, obj#6: [I
+- DeserializeToObject cast(value#2 as array<int>).toIntArray, obj#5: [I
+- LogicalRDD [value#2]
== Optimized Logical Plan ==
SerializeFromObject [newInstance(class
org.apache.spark.sql.catalyst.util.GenericArrayData) AS value#7]
+- MapElements <function1>, obj#6: [I
+- DeserializeToObject value#2.toIntArray, obj#5: [I
+- LogicalRDD [value#2]
== Physical Plan ==
*SerializeFromObject [newInstance(class
org.apache.spark.sql.catalyst.util.GenericArrayData) AS value#7]
+- *MapElements <function1>, obj#6: [I
+- *DeserializeToObject value#2.toIntArray, obj#5: [I
+- Scan ExistingRDD[value#2]
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
