Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/11722 )
Change subject: KUDU-2038: Support bitmap index ...................................................................... Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/11722/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11722/1//COMMIT_MSG@7 PS1, Line 7: KUDU-2038: Support bitmap indexing have you read the recent paper on Column Sketches? https://stratos.seas.harvard.edu/files/stratos/files/sketches.pdf It may be worth considering implementing this instead of bitmap indexes, with fewer of the downsides around memory usage. http://gerrit.cloudera.org:8080/#/c/11722/1//COMMIT_MSG@14 PS1, Line 14: The index could be created when creating or altering table, and : it could be dropped later. > The rows in a DRS is limited, so the memory for the bitmaps is limited too. We need to accumulate all of the values in memory before we can flush any of them, right? So, the memory consumption of a compaction becomes O(Size of all uncompressed unique values in indexed columns + scanner buffers) instead of O(scanner buffers), right? With a high number of indexed columns, with high cardinality, this could become significant. Should we consider some limits to automatically disable bitmap indexes if cardinality is too high? Or at least make sure this memory is appropriately tracked by MemTrackers so users understand the consumption and budget RAM appropriately? http://gerrit.cloudera.org:8080/#/c/11722/1//COMMIT_MSG@23 PS1, Line 23: The bitmap index supports Equality, InList, Range predicates. > I merged it from branch-1.4.x, and the benchmark is fascinating at the time There were a few other predicate evaluation perf bug fixes in the last 6-12 months, in particular KUDU-2231. Maybe you are seeing the improvement due to this change, and the bitmap indexes are not as helpful now that this bug is fixed? -- To view, visit http://gerrit.cloudera.org:8080/11722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0edaa0ef1dba2dbce85ebf15f0a731e4939a7860 Gerrit-Change-Number: 11722 Gerrit-PatchSet: 1 Gerrit-Owner: helifu <[email protected]> Gerrit-Reviewer: Andrew Wong <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Todd Lipcon <[email protected]> Gerrit-Reviewer: helifu <[email protected]> Gerrit-Comment-Date: Tue, 13 Nov 2018 23:47:15 +0000 Gerrit-HasComments: Yes
