[
https://issues.apache.org/jira/browse/IMPALA-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17637625#comment-17637625
]
Noemi Pap-Takacs commented on IMPALA-10001:
-------------------------------------------
To be precise, there is a flag 'enforce_sort_run_bytes_limit', which double
checks whether SORT_RUN_BYTES_LIMIT should really spill.
If the estimated sort data size is lower than the reservation, it predicts that
everything will fit into the memory and there is no need to spill. In this case
SORT_RUN_BYTES_LIMIT will be neglected and it will not force early spilling. If
spilling did happen because the estimate was too low,
'enforce_sort_run_bytes_limit' is set to true, and from that point
SORT_RUN_BYTES_LIMIT will regularly enforce spilling.
This means that with a good input size estimate SORT_RUN_BYTES_LIMIT can avoid
unnecessary spilling when the query can actually fit all data in memory.
However, when enforcing SORT_RUN_BYTES_LIMIT is necessary, the I/O overhead is
still present.
I think that Tim's idea in IMPALA-4530 would be the best solution to this
problem. He proposed "to produce many smaller in-memory runs, sort those with
quicksort, then do merge-sort" in-memory, instead of forming only one big run
or spilling many small runs.
> Find good default value for SORT_RUN_BYTES_LIMIT
> ------------------------------------------------
>
> Key: IMPALA-10001
> URL: https://issues.apache.org/jira/browse/IMPALA-10001
> Project: IMPALA
> Issue Type: Improvement
> Components: Perf Investigation
> Reporter: Riza Suminto
> Priority: Minor
>
> IMPALA-6692 add query option SORT_RUN_BYTES_LIMIT to trigger early sort
> before the query hit memory limit.
> Currently, it is disabled as default. We need to find a good default value
> for this query option.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
