Hello Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14804
to look at the new patch set (#2).
Change subject: IMPALA-9174: Speedup allocations of ORC Scanner
......................................................................
IMPALA-9174: Speedup allocations of ORC Scanner
The ORC library provides a hook for its clients to plugin a custom
memory pool. This memory pool needs to override the 'malloc()' and
'free()' methods. Impala has its own memory pool named OrcMemPool.
Impala's OrcMemPool used to have an internal unordered_map to keep
track of its allocations. In 'free()' it used the map to lookup the
size of the allocated byte array. We need this information in 'free()'
because of memory tracking. Therefore for each 'malloc()' and 'free()'
there was an additional allocation/deallocation by the unordered_map.
Instead of using a map this patch allocates a few more bytes on each
allocation and store the size of the allocated bytes in front of the
allocated bytes, similarly to how the standard malloc/free works.
(Unfortunately the standard malloc/free doesn't reveal that information
in a standard way).
OrcMemPool also had a method called 'FreeAll()' which freed all
allocated memory. This was a no-op because the library only uses the
memory pool to allocate memory for the data buffers, and they free their
memory in their destructors. On the other hand, it provided some kind of
guard against memory leaks in the ORC library. We can add some checks to
the destructor of OrcMemPool to detect leaks if we don't trust the
library's memory management.
Change-Id: Ia09e746883176d6f955c1718267bf55e2abb239b
---
M be/src/exec/hdfs-orc-scanner.cc
M be/src/exec/hdfs-orc-scanner.h
2 files changed, 11 insertions(+), 25 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/14804/2
--
To view, visit http://gerrit.cloudera.org:8080/14804
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia09e746883176d6f955c1718267bf55e2abb239b
Gerrit-Change-Number: 14804
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
