Koraseg opened a new pull request #30644: URL: https://github.com/apache/spark/pull/30644
**What changes were proposed in this pull request?** This PR introduces a specialized version of ArrayData class backed by primitive arrays. Then it is used for more efficient row decoding in case of non-nullable arrays. **Why are the changes needed?** Currently, even non-nullable arrays fields are represented as WrappedArray.ofRef in Row external representation. It leads to memory foorprint, as well as unnecessary cycles in boxing/unboxing operations. **Does this PR introduce any user-facing change?** UDFs depending on WrrappedArray.ofRef/WrappedArray[AnyRef] would stop working in case of non-nullable arrays. **How was this patch tested?** Pass the CIs. Also added a benchmark to illustrate the benefits. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
