Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18219 )
Change subject: IMPALA-11064 Optimizing Temporary File Structure for Batch Reading ...................................................................... Patch Set 6: (3 comments) http://gerrit.cloudera.org:8080/#/c/18219/5/be/src/exec/partitioned-hash-join-builder.h File be/src/exec/partitioned-hash-join-builder.h: http://gerrit.cloudera.org:8080/#/c/18219/5/be/src/exec/partitioned-hash-join-builder.h@80 PS5, Line 80: /// left shift 16 bits for the later specified partition id. > My understanding is that the partition number depends on the PARTITION_FANO Ack http://gerrit.cloudera.org:8080/#/c/18219/5/be/src/runtime/tmp-file-mgr.cc File be/src/runtime/tmp-file-mgr.cc: http://gerrit.cloudera.org:8080/#/c/18219/5/be/src/runtime/tmp-file-mgr.cc@130 PS5, Line 130: "If set to 0, means to use the same size as remote_tmp_file_size. " > The file and block size are power of 2 following the norm of the page size, Ack http://gerrit.cloudera.org:8080/#/c/18219/5/be/src/service/query-options.cc File be/src/service/query-options.cc: http://gerrit.cloudera.org:8080/#/c/18219/5/be/src/service/query-options.cc@1259 PS5, Line 1259: if (result != StringParser::PARSE_SUCCESS || level < -1 || level > 16) { > 2^12=4096, partition FANOUT by default is 16, if each block using 8MB, the I was mostly trying to understand the rationale for how it's set. If it were just packing everything into a 64-bit integer, that would allow for 2 32-bit ints. 12 seemed like a strange number because it doesn't align with any of the normal size variants (8/16/32/64). -- To view, visit http://gerrit.cloudera.org:8080/18219 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If913785cac9e2dafa20013b6600c87fcaf3e2018 Gerrit-Change-Number: 18219 Gerrit-PatchSet: 6 Gerrit-Owner: Yida Wu <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Yida Wu <[email protected]> Gerrit-Comment-Date: Wed, 29 Jun 2022 18:21:46 +0000 Gerrit-HasComments: Yes
