Internal Jenkins has submitted this change and it was merged. Change subject: IMPALA-3528: Transfer scratch tuple memory in Close() of Parquet scanner. ......................................................................
IMPALA-3528: Transfer scratch tuple memory in Close() of Parquet scanner. The lifetime of a scanner thread is decoupled from that of row batches that it produces. That means that all resources associated with row batches produced by the scanner thread should be transferred to those batches. The bug was that we were not transferring the ownership of memory from the scratch batch to the final row batch returned in HdfsParquetScanner::Close(). Triggering an event that would cause the freed memory to be dereferenced is possible, but very difficult. My understanding is that it is only possible in exceptional non-deterministic scenarios, e.g., a query is cancelled just at the right time, or the scanner hits a parse/decoding error. Testing: I tested this change locally by running the scanner and nested types test as well as TPCH, nested TPCH, and TPC-DS. Change-Id: Ic34d32c9a41ea66b2b2d8f5e187cc84d4cb569b2 Reviewed-on: http://gerrit.cloudera.org:8080/3041 Reviewed-by: Alex Behm <[email protected]> Tested-by: Internal Jenkins --- M be/src/exec/hdfs-parquet-scanner.cc 1 file changed, 2 insertions(+), 0 deletions(-) Approvals: Internal Jenkins: Verified Alex Behm: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/3041 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ic34d32c9a41ea66b2b2d8f5e187cc84d4cb569b2 Gerrit-PatchSet: 3 Gerrit-Project: Impala Gerrit-Branch: cdh5-trunk Gerrit-Owner: Alex Behm <[email protected]> Gerrit-Reviewer: Alex Behm <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Tim Armstrong <[email protected]>
