Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19601 Does it work for non-top-level array type fields and nested array? Generally I think this is not the right direction. The root cause is that, table cache array format is not the arrow-style format(put all leaf elements together) and can't fit in `ColumnarArray`. To avoid the expensive conversion, what you did here is kind of creating a fake `ColumnarArray` to be able to read the table cache array data directly. I think this is not a general solution but just a hack. Futher more, all complex types face the same problem in table cache. I think the right direction is to revisit the columnar cache format and design how to deal with complex types. This is something we dropped at the beginning(treat complext types as binary data) and now it's time to pay back.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org