Hello Tim Armstrong, Dan Hecht,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/3041
to look at the new patch set (#2).
Change subject: IMPALA-3528: Transfer scratch tuple memory in Close() of
Parquet scanner.
......................................................................
IMPALA-3528: Transfer scratch tuple memory in Close() of Parquet scanner.
The lifetime of a scanner thread is decoupled from that of row batches that
it produces. That means that all resources associated with row batches
produced by the scanner thread should be transferred to those batches.
The bug was that we were not transferring the ownership of memory from the
scratch batch to the final row batch returned in HdfsParquetScanner::Close().
Triggering an event that would cause the freed memory to be dereferenced is
possible, but very difficult. My understanding is that it is only possible
in exceptional non-deterministic scenarios, e.g., a query is cancelled just
at the right time, or the scanner hits a parse/decoding error.
Testing: I tested this change locally by running the scanner and nested
types test as well as TPCH, nested TPCH, and TPC-DS.
Change-Id: Ic34d32c9a41ea66b2b2d8f5e187cc84d4cb569b2
---
M be/src/exec/hdfs-parquet-scanner.cc
1 file changed, 2 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/41/3041/2
--
To view, visit http://gerrit.cloudera.org:8080/3041
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic34d32c9a41ea66b2b2d8f5e187cc84d4cb569b2
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Alex Behm <[email protected]>
Gerrit-Reviewer: Dan Hecht <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
