adriangb commented on PR #25576: URL: https://github.com/apache/datafusion/pull/25576#issuecomment-5766381627
A couple of questions: 1. Does the spec say this value is / must be `Exact` or could it be an estimate? An estimate sounds much better to me overall, it's just as useful for the planner and I don't think an exact value can be used for e.g. `select count(...) ...` if there is a filter or more than 1 row group or more than 1 file (which is a lot of queries in practice). 2. Could these values be derived from the dictionary size or bloom filters at read time? There's some interesting things that could be done e.g. you can even take the bloom filters, `OR` them and then derive an approximate distinct count for a single file with multiple row groups or even across files. You can't combine dictionary counts, but you could sum them or if willing to pay some cost upfront combine the dictionary hashes. 3. At write time could we do something similar and derive the value from the dictionary or bloom filters we are already writing, or prepare a bloom filter just to compute an estimated NDV? My estimate is that the current implementation uses ~9MB per column at the default row group size, a bloom filter would use ~1MB per column, would be cheaper to write and would have <1% error (this is all at the default NDV/FPP configuration) (not to mention it's free if you're already writing a bloom filter). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
