Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/18655
> For ArrowConverters, I thought we can skip the intermediate
ArrowRecordBatch creation in ArrowConverters.toPayloadIterator(). What do you
think about that?
Ok, I see. By using `ArrowWriter` directly on the root, then the
`ArrowFileWriter` can use that same root in creating the Byte array. So no
need to create an intermediate `ArrowRecordBatch`. That sounds good to me!
> For ColumnWriter, at first I'd like to support complex types like
ArrayType and StructType, so I refactored it based on your ColumnWriter
implementation. And then I renamed and moved the package
That's fine, but do they need to be in `o.a.s.sql.execution.vectorized`?
If so, then what's the point of having a `o.a.s.sql.execution.arrow` package if
`ArrowUtils` and `ArrowWriter` are not even there?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
