[
https://issues.apache.org/jira/browse/IMPALA-5824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-5824.
-----------------------------------
Resolution: Fixed
Fix Version/s: Impala 2.10.0
It looks like setting the mem_limit to 150mb (resulting in a buffer pool limit
of 75mb) is sufficient. The merge aggregation requires 114mb to run entirely
in memory so it should reliably spill.
{code}
+---------------------+--------+----------+----------+-------+------------+-----------+---------------+-----------------------+
| Operator | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows |
Peak Mem | Est. Peak Mem | Detail |
+---------------------+--------+----------+----------+-------+------------+-----------+---------------+-----------------------+
| 05:MERGING-EXCHANGE | 1 | 4.06s | 4.06s | 5 | 5 | 0 B
| 0 B | UNPARTITIONED |
| 02:TOP-N | 3 | 15.13ms | 15.30ms | 15 | 5 |
76.00 KB | 200 B | |
| 04:AGGREGATE | 3 | 2.32s | 2.34s | 1.50M | 1.56M |
114.16 MB | 34.00 MB | FINALIZE |
| 03:EXCHANGE | 3 | 1.51s | 1.53s | 1.50M | 1.56M | 0 B
| 0 B | HASH(l_orderkey) |
| 01:AGGREGATE | 3 | 923.22ms | 1.00s | 1.50M | 1.56M |
114.94 MB | 65.60 MB | STREAMING |
| 00:SCAN HDFS | 3 | 524.84ms | 596.76ms | 6.00M | 6.00M |
25.30 MB | 400.00 MB | tpch_parquet.lineitem |
+---------------------+--------+----------+----------+-------+------------+-----------+---------------+-----------------------+
{code}
> TestSpilling.test_spilling_aggs failing: SpilledPartitions: 0 (0)
> -----------------------------------------------------------------
>
> Key: IMPALA-5824
> URL: https://issues.apache.org/jira/browse/IMPALA-5824
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.10.0
> Reporter: David Knupp
> Assignee: Tim Armstrong
> Priority: Critical
> Labels: broken-build
> Fix For: Impala 2.10.0
>
>
> Recently seen in ASAN builds. The RUNTIME_PROFILE reports:
> {noformat}
> SpilledPartitions: 0 (0)
> {noformat}
> The test expects a result matching the following regex:
> {noformat}
> 02:35:30 TestSpilling.test_spilling_aggs[exec_option: {'debug_action': None,
> 'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
> 02:35:30 [gw3] linux2 -- Python 2.6.6
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/bin/../infra/python/env/bin/python
> 02:35:30 query_test/test_spilling.py:53: in test_spilling_aggs
> 02:35:30 self.run_test_case('QueryTest/spilling-aggs', vector)
> 02:35:30 common/impala_test_suite.py:430: in run_test_case
> 02:35:30 verify_runtime_profile(test_section['RUNTIME_PROFILE'],
> result.runtime_profile)
> 02:35:30 common/test_result_verifier.py:560: in verify_runtime_profile
> 02:35:30 actual))
> 02:35:30 E AssertionError: Did not find matches for lines in runtime
> profile:
> 02:35:30 E EXPECTED LINES:
> 02:35:30 E row_regex: .*SpilledPartitions: .* \([1-9][0-9]*\)
> {noformat}
> This has affected these variations of the test.
> {noformat}
> TestSpilling::()::test_spilling_aggs[exec_option: {'debug_action': None,
> 'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
> TestSpilling::()::test_spilling_aggs[exec_option: {'debug_action':
> '-1:OPEN:[email protected]',
> 'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
> TestSpilling::()::test_spilling_aggs[exec_option: {'debug_action':
> '-1:OPEN:[email protected]',
> 'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
> {noformat}
> Note that there were other failures in the same test runs similar to
> IMPALA-5820.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
