Jefffrey commented on code in PR #10596:
URL: https://github.com/apache/arrow-rs/pull/10596#discussion_r3744127085
##########
arrow/benches/cast_kernels.rs:
##########
@@ -202,6 +202,44 @@ fn build_nested_dict_array(size: usize) -> ArrayRef {
Arc::new(DictionaryArray::new(outer_keys, Arc::new(inner)))
}
+// Keys for a `size` row dictionary, spread over `distinct` values so a cast
has to touch
+// the whole values buffer rather than a contiguous prefix of it.
+fn dict_keys(size: usize, distinct: usize) -> UInt64Array {
+ let mut rng = seedable_rng();
+ let range = Uniform::new(0, distinct as u64).unwrap();
+ UInt64Array::from_iter_values((0..size).map(|_| rng.sample(range)))
+}
+
+// `Dictionary<UInt64, Utf8>` of `size` rows over `distinct` values, each
longer than 12
+// bytes so the resulting views reference the values buffer rather than
inlining.
+//
+// The ratio of rows to distinct values is what matters when casting to a
view: the cast
+// can either build one view per dictionary value and gather those with
`take`, or build
+// one view per row directly against the values buffer. Which of the two is
cheaper depends
+// on that ratio, so both a dense and a sparse shape are benchmarked below.
Review Comment:
dont have too much detail on implementation here; just keep simple like
"different implementation paths may be taken based on ratio of rows to distinct
values"
##########
arrow/benches/cast_kernels.rs:
##########
@@ -202,6 +202,44 @@ fn build_nested_dict_array(size: usize) -> ArrayRef {
Arc::new(DictionaryArray::new(outer_keys, Arc::new(inner)))
}
+// Keys for a `size` row dictionary, spread over `distinct` values so a cast
has to touch
+// the whole values buffer rather than a contiguous prefix of it.
+fn dict_keys(size: usize, distinct: usize) -> UInt64Array {
+ let mut rng = seedable_rng();
+ let range = Uniform::new(0, distinct as u64).unwrap();
+ UInt64Array::from_iter_values((0..size).map(|_| rng.sample(range)))
+}
+
+// `Dictionary<UInt64, Utf8>` of `size` rows over `distinct` values, each
longer than 12
+// bytes so the resulting views reference the values buffer rather than
inlining.
Review Comment:
should we have some mix of inline & long views to test more paths?
##########
arrow/benches/cast_kernels.rs:
##########
@@ -202,6 +202,44 @@ fn build_nested_dict_array(size: usize) -> ArrayRef {
Arc::new(DictionaryArray::new(outer_keys, Arc::new(inner)))
}
+// Keys for a `size` row dictionary, spread over `distinct` values so a cast
has to touch
+// the whole values buffer rather than a contiguous prefix of it.
+fn dict_keys(size: usize, distinct: usize) -> UInt64Array {
+ let mut rng = seedable_rng();
+ let range = Uniform::new(0, distinct as u64).unwrap();
+ UInt64Array::from_iter_values((0..size).map(|_| rng.sample(range)))
+}
+
+// `Dictionary<UInt64, Utf8>` of `size` rows over `distinct` values, each
longer than 12
+// bytes so the resulting views reference the values buffer rather than
inlining.
+//
+// The ratio of rows to distinct values is what matters when casting to a
view: the cast
+// can either build one view per dictionary value and gather those with
`take`, or build
+// one view per row directly against the values buffer. Which of the two is
cheaper depends
+// on that ratio, so both a dense and a sparse shape are benchmarked below.
+fn build_string_dict_array(size: usize, distinct: usize) -> ArrayRef {
+ let values =
+ StringArray::from_iter_values((0..distinct).map(|i|
format!("dictionary value {i:07}")));
+
+ Arc::new(DictionaryArray::new(
+ dict_keys(size, distinct),
+ Arc::new(values),
+ ))
+}
+
+// As `build_string_dict_array`, but with `Binary` values. Casting those to
`Utf8View` has
+// to validate the dictionary values as UTF-8, which a `Utf8` source does not.
+fn build_binary_dict_array(size: usize, distinct: usize) -> ArrayRef {
+ let values = BinaryArray::from_iter_values(
Review Comment:
could also get this by using output of `build_string_dict_array()` and
casting to `Dictionary(Binary)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
