lakshaydevoo-arch commented on code in PR #27008:
URL: https://github.com/apache/datafusion/pull/27008#discussion_r4489817328
##########
datafusion/common/benches/with_hashes.rs:
##########
@@ -178,8 +203,13 @@ fn create_null_mask(len: usize) -> NullBuffer
where
StandardUniform: Distribution<bool>,
{
+ create_null_mask_with_density(len, 0.03).expect("should be nulls in
buffer")
+}
+
+/// Returns a random null mask where each slot is null with probability
+/// `null_density`, or `None` if no slot ended up null
+fn create_null_mask_with_density(len: usize, null_density: f32) ->
Option<NullBuffer> {
let mut rng = make_rng();
- let null_density = 0.03;
let mut builder = NullBufferBuilder::new(len);
for _ in 0..len {
if rng.random::<f32>() < null_density {
Review Comment:
I've kept the masks intentionally nested across densities so that increasing
the density adds null positions while preserving the previous ones. I've
documented this behavior in create_null_mask. The mask generation now uses a
separate deterministic RNG stream from the dictionary data generation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
