[
https://issues.apache.org/jira/browse/IMPALA-7023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16479332#comment-16479332
]
ASF subversion and git services commented on IMPALA-7023:
---------------------------------------------------------
Commit 1e6544f7da1d756b437d8b0f12a6446f10f1f836 in impala's branch
refs/heads/master from [~joemcdonnell]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=1e6544f ]
IMPALA-7023: Wait for fragments to finish for test_insert.py
The arrangement of tests in test_insert.py changed with
IMPALA-7010, splitting out the memory limit tests into
test_insert_mem_limit(). On exhaustive, the combination
of test dimensions means test_insert_mem_limit() executes
11 different combinations. Each of these statements can
use a large amount of memory and this is not cleaned
up immediately. This has been causing
test_insert_overwrite(), which immediately follows
test_insert_mem_limit(), to hit the process memory limit.
This changes test_insert_mem_limit() to make it wait
for its fragments to finish.
Change-Id: I5642e9cb32dd02afd74dde7e0d3b31bddbee3ccd
Reviewed-on: http://gerrit.cloudera.org:8080/10426
Reviewed-by: Philip Zeyliger <phi...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
> TestInsertQueries.test_insert_overwrite fails by hitting memory limit
> ---------------------------------------------------------------------
>
> Key: IMPALA-7023
> URL: https://issues.apache.org/jira/browse/IMPALA-7023
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.13.0, Impala 3.1.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Blocker
> Labels: broken-build
>
> This failure is seen on exhaustive builds on both master and 2.x:
> {noformat}
> Error Message
> ImpalaBeeswaxException: ImpalaBeeswaxException: INNER EXCEPTION: <class
> 'beeswaxd.ttypes.BeeswaxException'> MESSAGE: AnalysisException: Failed to
> evaluate expr: 20 CAUSED BY: InternalException: Memory limit exceeded: Error
> occurred on backend
> impala-boost-static-burst-slave-el7-03d4.vpc.cloudera.com:22000 by fragment
> 0:0 Memory left in process limit: -4.29 GB
> Query(d24ea53242b4cedc:fc8e088500000000): Total=0 Peak=0 <unnamed>: Total=0
> Peak=0Process: memory limit exceeded. Limit=12.00 GB Total=16.29 GB
> Peak=16.29 GB Buffer Pool: Free Buffers: Total=160.00 KB Buffer Pool:
> Clean Pages: Total=0 Buffer Pool: Unused Reservation: Total=-328.00 KB
> Data Stream Service Queue: Limit=614.40 MB Total=0 Peak=116.12 KB Data
> Stream Manager Early RPCs: Total=0 Peak=6.76 KB TCMalloc Overhead:
> Total=103.56 MB RequestPool=fe-eval-exprs: Total=0 Peak=52.83 KB
> Query(d24ea53242b4cedc:fc8e088500000000): Total=0 Peak=0
> RequestPool=default-pool: Total=5.20 GB Peak=5.20 GB
> Query(f4014f7bb49ea78:6b926b1900000000): memory limit exceeded. Limit=64.00
> MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=1.76 GB Total=1.76 GB
> Peak=1.96 GB Query(2c44b65fbcb4e1ce:3d73badf00000000): memory limit
> exceeded. Limit=64.00 MB Reservation=0 ReservationLimit=32.00 MB
> OtherMemory=1.04 GB Total=1.04 GB Peak=1.61 GB
> Query(214cc23c1376176f:7844977b00000000): memory limit exceeded. Limit=64.00
> MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=1.23 GB Total=1.23 GB
> Peak=1.23 GB Query(8949bdf792a32ad2:33a36c0300000000): memory limit
> exceeded. Limit=64.00 MB Reservation=0 ReservationLimit=32.00 MB
> OtherMemory=642.20 MB Total=642.20 MB Peak=642.20 MB
> Query(5412ff4e6065721:519d3e6100000000): memory limit exceeded. Limit=64.00
> MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=556.49 MB Total=556.49
> MB Peak=556.49 MB Untracked Memory: Total=10.98 GB
> Stacktrace
> query_test/test_insert.py:132: in test_insert_overwrite
> multiple_impalad=vector.get_value('exec_option')['sync_ddl'] == 1)
> common/impala_test_suite.py:405: in run_test_case
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:620: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:173: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:339: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:335: in execute_query_async
> return self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:460: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
> E MESSAGE: AnalysisException: Failed to evaluate expr: 20
> E CAUSED BY: InternalException: Memory limit exceeded: Error occurred on
> backend impala-boost-static-burst-slave-el7-03d4.vpc.cloudera.com:22000 by
> fragment 0:0
> E Memory left in process limit: -4.29 GB
> E Query(d24ea53242b4cedc:fc8e088500000000): Total=0 Peak=0
> E <unnamed>: Total=0 Peak=0Process: memory limit exceeded. Limit=12.00 GB
> Total=16.29 GB Peak=16.29 GB
> E Buffer Pool: Free Buffers: Total=160.00 KB
> E Buffer Pool: Clean Pages: Total=0
> E Buffer Pool: Unused Reservation: Total=-328.00 KB
> E Data Stream Service Queue: Limit=614.40 MB Total=0 Peak=116.12 KB
> E Data Stream Manager Early RPCs: Total=0 Peak=6.76 KB
> E TCMalloc Overhead: Total=103.56 MB
> E RequestPool=fe-eval-exprs: Total=0 Peak=52.83 KB
> E Query(d24ea53242b4cedc:fc8e088500000000): Total=0 Peak=0
> E RequestPool=default-pool: Total=5.20 GB Peak=5.20 GB
> E Query(f4014f7bb49ea78:6b926b1900000000): memory limit exceeded.
> Limit=64.00 MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=1.76 GB
> Total=1.76 GB Peak=1.96 GB
> E Query(2c44b65fbcb4e1ce:3d73badf00000000): memory limit exceeded.
> Limit=64.00 MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=1.04 GB
> Total=1.04 GB Peak=1.61 GB
> E Query(214cc23c1376176f:7844977b00000000): memory limit exceeded.
> Limit=64.00 MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=1.23 GB
> Total=1.23 GB Peak=1.23 GB
> E Query(8949bdf792a32ad2:33a36c0300000000): memory limit exceeded.
> Limit=64.00 MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=642.20 MB
> Total=642.20 MB Peak=642.20 MB
> E Query(5412ff4e6065721:519d3e6100000000): memory limit exceeded.
> Limit=64.00 MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=556.49 MB
> Total=556.49 MB Peak=556.49 MB
> E Untracked Memory: Total=10.98 GB{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
