Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9002 )
Change subject: IMPALA-6383: free memory after skipping parquet row groups ...................................................................... Patch Set 1: (4 comments) http://gerrit.cloudera.org:8080/#/c/9002/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/9002/1//COMMIT_MSG@13 PS1, Line 13: Testing: My feeling is that this testing is enough for now. With the reservation changes forthcoming, the multiple-row-group tests failed hard when it tried to allocate new buffers for the next row group when the buffers for the previous row group were released. We could add a mem_limit test tuned to reproduce it but those have a tendency to be flaky and it doesn't seem worth the effort when more robust test coverage will be forthcoming. http://gerrit.cloudera.org:8080/#/c/9002/1/be/src/exec/hdfs-parquet-scanner.h File be/src/exec/hdfs-parquet-scanner.h: http://gerrit.cloudera.org:8080/#/c/9002/1/be/src/exec/hdfs-parquet-scanner.h@647 PS1, Line 647: /// were returned. and when returning the last batch. > nit: remove "."? Done http://gerrit.cloudera.org:8080/#/c/9002/1/be/src/exec/hdfs-parquet-scanner.cc File be/src/exec/hdfs-parquet-scanner.cc: http://gerrit.cloudera.org:8080/#/c/9002/1/be/src/exec/hdfs-parquet-scanner.cc@623 PS1, Line 623: // We skipped the previous row group. Release resources before starting another. > This gets called for cases where no resources have been reserved (e.g. L655 I could also reorganise the code so that ReleaseSkippedRowGroupResources() gets called before "continue" so that it's more explicit. I was originally thinking that would be error-prone and easy to forget, but with the NumStreams() DCHECK maybe that won't be the case. http://gerrit.cloudera.org:8080/#/c/9002/1/be/src/exec/hdfs-parquet-scanner.cc@738 PS1, Line 738: col_reader->Close(row_batch); > nit: while you're here you could reduce this to a single line. Done -- To view, visit http://gerrit.cloudera.org:8080/9002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc2f8f27c9b238be60261539f8d4be2facb57a2b Gerrit-Change-Number: 9002 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Lars Volker <[email protected]> Gerrit-Reviewer: Michael Ho <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Comment-Date: Thu, 11 Jan 2018 19:16:43 +0000 Gerrit-HasComments: Yes
