Rich-T-kid commented on PR #25576: URL: https://github.com/apache/datafusion/pull/25576#issuecomment-5766705291
1. No, from my understanding it doesn't need to be exact. Even if it were, it wouldn't be any more useful than an estimation, for the reasons you mentioned. A parquet file with two row groups with an NDV of 100 and 101 respectively can have between 101 and 201 unique values. 2. for bloom filters I think this is a good idea but from the arrow spec, dictionary arrays aren't guaranteed to be unique. ``` dictionary = keys:[0,1,2,3] ,values=["cat","cat","cat","cat"] ``` is a valid dictionary array. I think the bloom filter idea is the best approach to derive the ndv as its pretty expensive to get an exact count. With that being said, many work loads are write once read thousands of times so the trade off in practice is generally worth it. especially if we can take advantage of low ndv columns (👀 #24111) 3. +1 with this. i'm actually about to open up a PR that adds benchmarks for calculating the ndv. it may be interesting to open an issue with possible optimization ideas. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
