Rich-T-kid commented on code in PR #10765:
URL: https://github.com/apache/arrow-rs/pull/10765#discussion_r3818374976
##########
arrow-array/src/builder/generic_bytes_dictionary_builder.rs:
##########
@@ -377,6 +377,95 @@ where
}
}
+ /// Append all values from a [`GenericByteArray`] into this builder.
+ ///
+ /// This is more efficient than calling [`Self::append`] in a loop because
+ /// it accesses the raw offset/value buffers directly and bulk-writes the
+ /// resolved keys with a single `append_slice` / `append_values` call.
+ ///
+ /// Returns an error if any new dictionary index would overflow the key
type.
+ pub fn append_array(&mut self, array: &GenericByteArray<T>) -> Result<(),
ArrowError> {
+ let row_count = array.len();
+ if row_count == 0 {
+ return Ok(());
+ }
+ let offsets = array.value_offsets();
+ let raw_data = array.value_data();
+
+ match array.nulls() {
+ None => {
+ let mut key_buf: Vec<K::Native> =
Vec::with_capacity(row_count);
+ for row_idx in 0..row_count {
+ let start = offsets[row_idx].as_usize();
+ let end = offsets[row_idx + 1].as_usize();
+ // SAFETY: offsets are valid by GenericByteArray invariants
+ let bytes = unsafe { raw_data.get_unchecked(start..end) };
Review Comment:
https://github.com/apache/arrow-rs/pull/10765/commits/7be8b2107c8257339c7307b8dfed82fa4704ed0b
does just that
##########
arrow-array/src/builder/generic_bytes_dictionary_builder.rs:
##########
@@ -377,6 +377,95 @@ where
}
}
+ /// Append all values from a [`GenericByteArray`] into this builder.
+ ///
+ /// This is more efficient than calling [`Self::append`] in a loop because
+ /// it accesses the raw offset/value buffers directly and bulk-writes the
+ /// resolved keys with a single `append_slice` / `append_values` call.
+ ///
+ /// Returns an error if any new dictionary index would overflow the key
type.
+ pub fn append_array(&mut self, array: &GenericByteArray<T>) -> Result<(),
ArrowError> {
+ let row_count = array.len();
+ if row_count == 0 {
+ return Ok(());
+ }
+ let offsets = array.value_offsets();
+ let raw_data = array.value_data();
+
+ match array.nulls() {
+ None => {
+ let mut key_buf: Vec<K::Native> =
Vec::with_capacity(row_count);
+ for row_idx in 0..row_count {
+ let start = offsets[row_idx].as_usize();
+ let end = offsets[row_idx + 1].as_usize();
+ // SAFETY: offsets are valid by GenericByteArray invariants
+ let bytes = unsafe { raw_data.get_unchecked(start..end) };
Review Comment:
I think placing this in a anon func would be the best way to solve this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
