[
https://issues.apache.org/jira/browse/IMPALA-6604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-6604:
----------------------------------
Component/s: Infrastructure
> rethink stress test binary search cutoff point
> ----------------------------------------------
>
> Key: IMPALA-6604
> URL: https://issues.apache.org/jira/browse/IMPALA-6604
> Project: IMPALA
> Issue Type: Task
> Components: Infrastructure
> Affects Versions: Impala 2.11.0
> Reporter: Michael Brown
> Priority: Major
>
> While working on IMPALA-5721 I see that during a binary search, a lot of
> queries that are meant to be tested with spilling are not set at any
> different mem_limit than they were for their non-spillable counterparts. This
> is due to both the default data scales of the stress test (TPCH 100, TPCDS
> 300) and the fact that {{MEM_LIMIT_EQ_THRESHOLD_MB = 50}} in
> {{concurrent_select.py}}. For queries that only use roughly 200 MB, a 50MB
> threshold is a rather large percentage. This means the binary search is
> "giving up" sooner than it could.
> There are some options:
> - drop MEM_LIMIT_EQ_THRESHOLD_MB completely and only rely on
> MEM_LIMIT_EQ_THRESHOLD_PC, which is 0.975. For a 200 MB query, this would
> effectively be a threshold of 5MB. This would lead to a more precise binary
> search, but one that would take longer to complete.
> - keep MEM_LIMIT_EQ_THRESHOLD_MB but only put it in play for either larger
> queries or larger data sets.
> - do not test at these data scales at all and instead move toward testing at
> data scales of, say, TPCH 1000
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
