Tamas Mate created IMPALA-11690:
-----------------------------------
Summary: Estimate codegen memory usage
Key: IMPALA-11690
URL: https://issues.apache.org/jira/browse/IMPALA-11690
Project: IMPALA
Issue Type: Improvement
Components: Backend, Frontend
Affects Versions: Impala 4.1.0
Reporter: Tamas Mate
Codegen memory requirement is estimated only on the backend before allocation:
{code:c++}
int64_t estimated_memory = ESTIMATED_OPTIMIZER_BYTES_PER_INST
* counter.GetCount(InstructionCounter::TOTAL_INSTS);
if (!mem_tracker_->TryConsume(estimated_memory)) {
const string& msg = Substitute(
"Codegen failed to reserve '$0' bytes for optimization",
estimated_memory);
return mem_tracker_->MemLimitExceeded(NULL, msg, estimated_memory);
}
{code}
This can cause problems with dedicated coordinator memory estimates where the
memory is more tightly estimated, for example:
{code:none}Query(604397725bdbccbb:8fc7624b00000000): Limit=131.00 MB
Reservation=13.94 MB ReservationLimit=99.00 MB OtherMemory=399.55 KB
Total=14.33 MB Peak=14.33 MB
Unclaimed reservations: Reservation=13.94 MB OtherMemory=0 Total=13.94 MB
Peak=13.94 MB
...
Dedicated Coordinator Resource Estimate: Memory=131MB
{code}
Impala could apply some heuristics for the codgen memory requirements as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
