Yida Wu has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/16638 )
Change subject: IMPALA-10102 Fix Impalad crashes when writing a parquet file with large rows. ...................................................................... IMPALA-10102 Fix Impalad crashes when writing a parquet file with large rows. The crash happens when trying to dereference a null pointer returned from a failed memory allocation from memory pool. TryAllocate is used instead of Allocate and null check is added for the large memory allocations such as buffer for dictionary page and compressed dictionary page. The memory allocation is most likely to fail for these large allocations when memory is scarce. This change fixes the crash in this particular code path, however in practice, there could still be an OOM issue which could lead to the process getting killed by the OS. The change doesn't fix the OOM issue, users need to configure the mem_limit (start-up option) properly to avoid the OOM crash. Test: Ran a script to redo the test mentioned in the Jira for thirty times, no crash happens. Change-Id: I0dee474cceb0c370278d290eb900c05769b23dec --- M be/src/exec/parquet/hdfs-parquet-table-writer.cc 1 file changed, 19 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/16638/3 -- To view, visit http://gerrit.cloudera.org:8080/16638 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0dee474cceb0c370278d290eb900c05769b23dec Gerrit-Change-Number: 16638 Gerrit-PatchSet: 3 Gerrit-Owner: Yida Wu <[email protected]> Gerrit-Reviewer: Abhishek Rawat <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
