marvinlanhenke commented on issue #10806:
URL: https://github.com/apache/datafusion/issues/10806#issuecomment-2156415596
@alamb
...while prototyping I ran into an "issue" or some questions, perhaps you
can help here.
Since we could have more than one page per column per row-group, I
implemented the e.g. `min` statistics as an Iterator to yield the global
minimum across all pages. First of all does this even make sense here?
```Rust
impl<'a, I> Iterator for MinInt64PageStatsIterator<'a, I>
where
I: Iterator<Item = Option<&'a Index>>,
{
type Item = Option<i64>;
fn next(&mut self) -> Option<Self::Item> {
let next = self.iter.next();
next.flatten().map(|index| match index {
Index::INT64(native_idx) => native_idx
.indexes
.iter()
.flat_map(|page_idx| page_idx.min)
.min(),
_ => None,
})
}
fn size_hint(&self) -> (usize, Option<usize>) {
self.iter.size_hint()
}
}
```
If it does make sense to "coalesce" into a global page minimum, I'll run
into trouble later on when I create those iterators for `f32, f64` since the
trait `Ord` is not implemented. We could workaround this by using
`reduce(f32::min)` - but I want to check if I'm on the right track at all - or
perhaps I should coalesce the page minimum into a global?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
