Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11582 )
Change subject: IMPALA-6658: improve Parquet RLE for low bit widths ...................................................................... Patch Set 4: (4 comments) I went through code again and my impression is that MaxBufferSize does not do its job properly. http://gerrit.cloudera.org:8080/#/c/11582/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11582/2//COMMIT_MSG@20 PS2, Line 20: length 16 for single bit values. All other bit widths will use the : existing length 8 runs. : > I agree that this might be a better way, but I am happy with the simple imp Ok, the biggest win (50% size for alternating runs of 8 in the 1 bit case) is already gained with this change. Please reflect in the Jira that there is still some potential in improving the RLE encoding. http://gerrit.cloudera.org:8080/#/c/11582/4/be/src/util/rle-encoding.h File be/src/util/rle-encoding.h: http://gerrit.cloudera.org:8080/#/c/11582/4/be/src/util/rle-encoding.h@250 PS4, Line 250: 1 + Can you simplify this expression? MAX_VALUES_PER_LITERAL_RUN must be divisible by 8 - this could be checked by an assert, and the Ceil + static cast could be removed. http://gerrit.cloudera.org:8080/#/c/11582/4/be/src/util/rle-encoding.h@264 PS4, Line 264: num_runs Aren't we double counting the indicator byte here? My assumption is that this num_runs is meant to add the +1 byte/ literal run. http://gerrit.cloudera.org:8080/#/c/11582/4/be/src/util/rle-encoding.h@265 PS4, Line 265: return std::max(MinBufferSize(bit_width), literal_max_size); This seems to assume that a single big literal run is the worsts case - can't this underestimate the number of bytes needed due to IMPALA-6658? -- To view, visit http://gerrit.cloudera.org:8080/11582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I191a581d3f699b6669e48ac9dc39c76ed77c4a76 Gerrit-Change-Number: 11582 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman <[email protected]> Gerrit-Reviewer: Andrew Sherman <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Thomas Marshall <[email protected]> Gerrit-Comment-Date: Tue, 30 Oct 2018 15:55:19 +0000 Gerrit-HasComments: Yes
