Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/11582 )
Change subject: IMPALA-6658: improve Parquet RLE for low bit widths ...................................................................... Patch Set 2: (4 comments) Thanks Csaba http://gerrit.cloudera.org:8080/#/c/11582/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11582/2//COMMIT_MSG@20 PS2, Line 20: By default RleEncoder will now use run length encoding for runs of : length 24 for single bit values, and of length 16 for 2 bit wide values. : All other bit widths will use the existing length 8 runs. > Ok, the biggest win (50% size for alternating runs of 8 in the 1 bit case) So you want me to write something like this in the jira? If you know the structure of the data then better encodings are possible. For example with bit_width=1, using min_run_length=24 is better in the case where we avoid interrupting a literal run. Using the notation of 'RXX' for a repeated run of length XX (so R16 is a run of lngth 16), and 'LYY' for a literal run of length YY. L24 R16 L24 R16 L24 min_run_length 8 4 2 4 2 4 min_run_length 16 (new default) 4 2 4 2 4 min_run_length 24 4 2 3 2 3 (one long literal run) So it is possible to optimize by detecting this situation and avoiding breaking a long literal run for a run of length 16. http://gerrit.cloudera.org:8080/#/c/11582/4/be/src/util/rle-encoding.h File be/src/util/rle-encoding.h: http://gerrit.cloudera.org:8080/#/c/11582/4/be/src/util/rle-encoding.h@250 PS4, Line 250: iter > Can you simplify this expression? MAX_VALUES_PER_LITERAL_RUN must be divisi Good idea http://gerrit.cloudera.org:8080/#/c/11582/4/be/src/util/rle-encoding.h@264 PS4, Line 264: insertin > Aren't we double counting the indicator byte here? My assumption is that th I think this is correct as is but I am changing it to the clearer literal_max_size = num_runs * (1 + bytes_per_run) -- 1 is the indicator -- bytes_per_run is the encoded bytes http://gerrit.cloudera.org:8080/#/c/11582/4/be/src/util/rle-encoding.h@265 PS4, Line 265: / 1 byte for the encoded run length > This seems to assume that a single big literal run is the worsts case - can I think before this change a long literal was not always the worst case, but now it is. -- To view, visit http://gerrit.cloudera.org:8080/11582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I191a581d3f699b6669e48ac9dc39c76ed77c4a76 Gerrit-Change-Number: 11582 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman <[email protected]> Gerrit-Reviewer: Andrew Sherman <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Thomas Marshall <[email protected]> Gerrit-Comment-Date: Tue, 30 Oct 2018 22:22:19 +0000 Gerrit-HasComments: Yes
