Rich-T-kid commented on issue #8608: URL: https://github.com/apache/arrow-rs/issues/8608#issuecomment-5262298172
> I wonder if rather than populating `distinct_count` we could instead add an approximate count somewhere using a sketch of some sort (https://datasketches.apache.org/). If all you're after is relative sizes for query planning I'd think a sketch would suffice. 🤔 I think this could be a good idea but as @alamb mentioned, for correcntess reasons we cant do this. its worth noting the `distinct_count` field is already an estimation atleast for how we intend to use it for datafusion. the field is set on a per row-group basis with no guarentees for its relationship to other columns. The same column in two different row groups can each report a `distinct_count` of 100, but consumers can't assume the combined cardinality, it could be 100 (full overlap), 200 (no overlap), or anything in between. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
