Alex Behm has uploaded a new change for review. http://gerrit.cloudera.org:8080/3041
Change subject: IMPALA-3528: Transfer scratch tuple memory in Close() of Parquet scanner. ...................................................................... IMPALA-3528: Transfer scratch tuple memory in Close() of Parquet scanner. The lifetime of a scanner thread is decoupled from that of row batches that it produces. That means that all resources associated with row batches produced by the scanner thread should be transferred to those batches. The bug was that we were not transferring the ownership of memory from the scratch batch to the final row batch returned in HdfsParquetScanner::Close(). Triggering an event that would cause the freed memory to be dereferenced is possible, but very difficult. My understanding is that it is only possible in exceptional non-deterministic scenarios, e.g., a query is cancelled just at the right time, or the scanner hits a parse/decoding error. Testing: I tested this change locally by running the scanner and nested types test as well as TPCH, nested TPCH, and TPC-DS. Change-Id: Ic34d32c9a41ea66b2b2d8f5e187cc84d4cb569b2 --- M be/src/exec/hdfs-parquet-scanner.cc 1 file changed, 2 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/41/3041/1 -- To view, visit http://gerrit.cloudera.org:8080/3041 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ic34d32c9a41ea66b2b2d8f5e187cc84d4cb569b2 Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: cdh5-trunk Gerrit-Owner: Alex Behm <[email protected]>
