Jason Fehr has uploaded a new patch set (#21). ( http://gerrit.cloudera.org:8080/24159 )
Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing ...................................................................... IMPALA-14863: Fixes Potential OOM During Row Batch Processing Impala checks at the end of every row batch to determine if the query or process memory limit has been exceeded. If the batch processing requires evaluating expressions that allocate memory for their results (such as UPPER() or LOWER()), that memory is allocated without checking any memory limits. If a large number of these expression evaluations are performed during a single row batch, memory will be consumed until the process memory limit is reached. At that point, the executor begins failing other queries with cannot allocate memory errors as they request memory. However, the row batch processing continues to run since it does no memory limit checking during batch processing. This patch determines a limit for the amount of memory that conjunct expression evaluation results can consume as a percentage of the lowest soft memory limit for the query execution. Before evaluating conjuncts for each join probe row, the expression results memory pool size is cleared if its allocated memory has exceeded this threshold. Clearing the memory pool does not result in memory being freed but instead marks all memory in the pool as available for re-use. Testing accomplished by a new custom cluster test that replicates the query that revealed this issue. Regression functionality testing performed by running existing test suite. Regression performance testing performed by running AB test job. Generated-by: Github Copilot (GPT-5.3-Codex) Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af --- M be/src/codegen/llvm-codegen-test.cc M be/src/codegen/llvm-codegen.cc M be/src/codegen/llvm-codegen.h M be/src/exec/partitioned-hash-join-node-ir.cc M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/exec/partitioned-hash-join-node.inline.h A tests/custom_cluster/test_conjuncts.py 8 files changed, 251 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/24159/21 -- To view, visit http://gerrit.cloudera.org:8080/24159 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af Gerrit-Change-Number: 24159 Gerrit-PatchSet: 21 Gerrit-Owner: Jason Fehr <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Jason Fehr <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]>
