okhsunrog commented on code in PR #10927:
URL: https://github.com/apache/arrow-rs/pull/10927#discussion_r3905615782
##########
arrow-select/src/dictionary.rs:
##########
@@ -380,6 +464,58 @@ mod tests {
use arrow_buffer::{BooleanBuffer, Buffer, NullBuffer, OffsetBuffer};
use std::sync::Arc;
+ use arrow_array::types::UInt16Type;
+ use arrow_array::{StringViewArray, UInt16Array};
+
+ #[test]
+ #[cfg_attr(miri, ignore)] // Takes too long
+ fn merge_string_view_dictionaries_deduplicates_exactly() {
+ // Four dictionaries over the same values: 60000 distinct strings, an
+ // empty string and a null. Concatenating them would need 240008 keys,
+ // far past the UInt16 range, while the distinct values leave room to
+ // spare -- so the merge has to deduplicate them exactly. At this
+ // cardinality the best-effort interner alone leaves thousands of
+ // duplicates behind and overflows, which forces the exact retry.
+ const DISTINCT: usize = 60000;
Review Comment:
Good call, done. `u8` keys over 200 distinct values still leaves the
interner with more duplicates than 256 keys can address, so the exact retry is
still the path under test. The miri skip is gone and the test runs in about 14
seconds there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
