Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14804
Change subject: IMPALA-9174: Speedup allocations of ORC Scanner ...................................................................... IMPALA-9174: Speedup allocations of ORC Scanner The ORC library provides a hook for its clients to plugin a custom memory pool. This memory pool needs to override the 'malloc()' and 'free()' methods. Impala has its own memory pool named OrcMemPool. Impala's OrcMemPool used to have an internal unordered_map to keep track of its allocations. In 'free()' it used the map to lookup the size of the allocated byte array. We need this information in 'free()' because of memory tracking. Therefore for each 'malloc()' and 'free()' there was an additional allocation/deallocation by the unordered_map. Instead of using a map this patch allocates a few more bytes on each allocation and store the size of the allocated bytes in front of the allocated bytes, similarly to how the standard malloc/free works. (Unfortunately the standard malloc/free doesn't reveal that information in a standard way). OrcMemPool also had a method called 'FreeAll()' which freed all allocated memory. This was a no-op because the library only uses the memory pool to allocate memory for the data buffers, and they free their memory in their destructors. On the other hand, it provided some kind of guard against memory leaks in the ORC library. We can add some checks to the destructor of OrcMemPool to detect leaks if we don't trust the library's memory management. Change-Id: Ia09e746883176d6f955c1718267bf55e2abb239b --- M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-orc-scanner.h 2 files changed, 7 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/14804/1 -- To view, visit http://gerrit.cloudera.org:8080/14804 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia09e746883176d6f955c1718267bf55e2abb239b Gerrit-Change-Number: 14804 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
