Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19601
Does it work for non-top-level array type fields and nested array?
Generally I think this is not the right direction. The root cause is that,
table cache array format is not the arrow-style format(put all leaf elements
together) and can't fit in `ColumnarArray`. To avoid the expensive conversion,
what you did here is kind of creating a fake `ColumnarArray` to be able to read
the table cache array data directly. I think this is not a general solution but
just a hack.
Futher more, all complex types face the same problem in table cache. I
think the right direction is to revisit the columnar cache format and design
how to deal with complex types. This is something we dropped at the
beginning(treat complext types as binary data) and now it's time to pay back.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
