n3nash commented on issue #666: Add support for dynamic bloom filter to increase efficiency of bloom filter for static sizing URL: https://github.com/apache/incubator-hudi/pull/666#issuecomment-489899078 I did not see any significant increase in size on disk, if you see my comment above on performance results, there is one on size and both of them are close to 2MB, I actually rounded them off to the near megabyte, there may be differences in kilobytes. Did not capture for reading times (disk -> memory), I'll add some results for that too. But I was thinking of bumping up the default entries higher as well, to 500Kish.. General comment : Option1 : Should we just use DynamicBloom by default ? What we can do is add a bit in the footer to say if the new bloom filter is dynamic or not. When we deserialize, if we find this bit, we treat it as dynamic, else regular bloom. Whenever a file is rewritten (through merge handle), convert the regular bloom to dynamic.. Option 2 : Start with a large default for bloom filter (assuming read is not affected much). If the file is ever rewritten, depending on the number of entries in the parquet file, reevaluate the number of entries in the bloom filter and reset it..
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
