Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8707 )
Change subject: IMPALA-4835: Part 2: Allocate scan range buffers upfront ...................................................................... Patch Set 24: (1 comment) http://gerrit.cloudera.org:8080/#/c/8707/24/be/src/runtime/io/disk-io-mgr.h File be/src/runtime/io/disk-io-mgr.h: http://gerrit.cloudera.org:8080/#/c/8707/24/be/src/runtime/io/disk-io-mgr.h@308 PS24, Line 308: int64_t max_bytes); > I'm not arguing for one way or the other at this point. I'm just trying to The code before this patch limited the number of queued buffers rather than the total number of buffers. I.e. it didn't factor in buffers in the client so couldn't control total reservation consumption. So there are some changes needed to correctly account for that and unblock the ranges in ReturnBuffer() instead of GetNext(). Giving each scanner a reservation for 3 * buffer size doesn't work for columnar formats because there's a scan range per column rather than per scanner. The commit message for Part 3 summarises the challenge there. Given a reservation of 3 * 8MB * num columns would result in really huge reservations for wide parquet files. We want to reserve less than that to avoid bad regressions. That then results in some interesting cases where we have to divide some amount of memory between many irregular-sized small scan ranges for the different Parquet columns where we want to use the available reservation efficiently. -- To view, visit http://gerrit.cloudera.org:8080/8707 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia243bf8b62feeea602b122e0503ea4ba7d3ee70f Gerrit-Change-Number: 8707 Gerrit-PatchSet: 24 Gerrit-Owner: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Reviewer: Bikramjeet Vig <bikramjeet....@cloudera.com> Gerrit-Reviewer: Dan Hecht <dhe...@cloudera.com> Gerrit-Reviewer: Michael Ho <k...@cloudera.com> Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Comment-Date: Tue, 13 Feb 2018 22:34:57 +0000 Gerrit-HasComments: Yes
