lakshaydevoo-arch commented on code in PR #27008:
URL: https://github.com/apache/datafusion/pull/27008#discussion_r4489855886
##########
datafusion/common/benches/with_hashes.rs:
##########
@@ -178,8 +203,13 @@ fn create_null_mask(len: usize) -> NullBuffer
where
StandardUniform: Distribution<bool>,
{
+ create_null_mask_with_density(len, 0.03).expect("should be nulls in
buffer")
+}
+
+/// Returns a random null mask where each slot is null with probability
+/// `null_density`, or `None` if no slot ended up null
+fn create_null_mask_with_density(len: usize, null_density: f32) ->
Option<NullBuffer> {
let mut rng = make_rng();
Review Comment:
found a strong correlation between the selected dictionary values and null
positions. switched the null mask rng to a separate seed (43) and verified the
distribution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
