[ 
https://issues.apache.org/jira/browse/IMPALA-7023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16479403#comment-16479403
 ] 

Joe McDonnell commented on IMPALA-7023:
---------------------------------------

[~tarmstrong] I think it is distinct from IMPALA-2990, and we should confirm 
that the behavior is what we expect. The actual time the fragments stick around 
is not extremely long. In a recent run, the first test_insert_mem_limit() test 
runs at 19:18:06, and there are 11 of them. The first test_insert_overwrite() 
test runs a minute later at 19:19:04. When it fails, there are three queries 
around:
{noformat}
       Query(4c44d3a9543363b6:23c9f0800000000): memory limit exceeded. 
Limit=64.00 MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=479.82 MB 
Total=479.82 MB Peak=1.96 GB
E       Query(644c5eae5958c33e:83caf2a500000000): memory limit exceeded. 
Limit=64.00 MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=1.28 GB 
Total=1.28 GB Peak=1.28 GB
E       Query(1f414583bd506a20:d785ed1c00000000): memory limit exceeded. 
Limit=64.00 MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=984.86 MB 
Total=984.86 MB Peak=984.86 MB
E       Query(324b13762c89d8da:6bd4d6f800000000): Total=0 Peak=0
E     Untracked Memory: Total=12.08 GB{noformat}
The test_insert_overwrite() tests stop failing at 19:19:42. Given this timing, 
this behavior could have been happening for a while without us noticing. I will 
file a follow-up JIRA.

> TestInsertQueries.test_insert_overwrite fails by hitting memory limit
> ---------------------------------------------------------------------
>
>                 Key: IMPALA-7023
>                 URL: https://issues.apache.org/jira/browse/IMPALA-7023
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>    Affects Versions: Impala 2.13.0, Impala 3.1.0
>            Reporter: Joe McDonnell
>            Assignee: Joe McDonnell
>            Priority: Blocker
>              Labels: broken-build
>
> This failure is seen on exhaustive builds on both master and 2.x:
> {noformat}
> Error Message
> ImpalaBeeswaxException: ImpalaBeeswaxException:  INNER EXCEPTION: <class 
> 'beeswaxd.ttypes.BeeswaxException'>  MESSAGE: AnalysisException: Failed to 
> evaluate expr: 20 CAUSED BY: InternalException: Memory limit exceeded: Error 
> occurred on backend 
> impala-boost-static-burst-slave-el7-03d4.vpc.cloudera.com:22000 by fragment 
> 0:0 Memory left in process limit: -4.29 GB 
> Query(d24ea53242b4cedc:fc8e088500000000): Total=0 Peak=0   <unnamed>: Total=0 
> Peak=0Process: memory limit exceeded. Limit=12.00 GB Total=16.29 GB 
> Peak=16.29 GB   Buffer Pool: Free Buffers: Total=160.00 KB   Buffer Pool: 
> Clean Pages: Total=0   Buffer Pool: Unused Reservation: Total=-328.00 KB   
> Data Stream Service Queue: Limit=614.40 MB Total=0 Peak=116.12 KB   Data 
> Stream Manager Early RPCs: Total=0 Peak=6.76 KB   TCMalloc Overhead: 
> Total=103.56 MB   RequestPool=fe-eval-exprs: Total=0 Peak=52.83 KB     
> Query(d24ea53242b4cedc:fc8e088500000000): Total=0 Peak=0   
> RequestPool=default-pool: Total=5.20 GB Peak=5.20 GB     
> Query(f4014f7bb49ea78:6b926b1900000000): memory limit exceeded. Limit=64.00 
> MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=1.76 GB Total=1.76 GB 
> Peak=1.96 GB     Query(2c44b65fbcb4e1ce:3d73badf00000000): memory limit 
> exceeded. Limit=64.00 MB Reservation=0 ReservationLimit=32.00 MB 
> OtherMemory=1.04 GB Total=1.04 GB Peak=1.61 GB     
> Query(214cc23c1376176f:7844977b00000000): memory limit exceeded. Limit=64.00 
> MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=1.23 GB Total=1.23 GB 
> Peak=1.23 GB     Query(8949bdf792a32ad2:33a36c0300000000): memory limit 
> exceeded. Limit=64.00 MB Reservation=0 ReservationLimit=32.00 MB 
> OtherMemory=642.20 MB Total=642.20 MB Peak=642.20 MB     
> Query(5412ff4e6065721:519d3e6100000000): memory limit exceeded. Limit=64.00 
> MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=556.49 MB Total=556.49 
> MB Peak=556.49 MB   Untracked Memory: Total=10.98 GB
> Stacktrace
> query_test/test_insert.py:132: in test_insert_overwrite
>     multiple_impalad=vector.get_value('exec_option')['sync_ddl'] == 1)
> common/impala_test_suite.py:405: in run_test_case
>     result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:620: in __execute_query
>     return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute
>     return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:173: in execute
>     handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:339: in __execute_query
>     handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:335: in execute_query_async
>     return self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:460: in __do_rpc
>     raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E   ImpalaBeeswaxException: ImpalaBeeswaxException:
> E    INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
> E    MESSAGE: AnalysisException: Failed to evaluate expr: 20
> E   CAUSED BY: InternalException: Memory limit exceeded: Error occurred on 
> backend impala-boost-static-burst-slave-el7-03d4.vpc.cloudera.com:22000 by 
> fragment 0:0
> E   Memory left in process limit: -4.29 GB
> E   Query(d24ea53242b4cedc:fc8e088500000000): Total=0 Peak=0
> E     <unnamed>: Total=0 Peak=0Process: memory limit exceeded. Limit=12.00 GB 
> Total=16.29 GB Peak=16.29 GB
> E     Buffer Pool: Free Buffers: Total=160.00 KB
> E     Buffer Pool: Clean Pages: Total=0
> E     Buffer Pool: Unused Reservation: Total=-328.00 KB
> E     Data Stream Service Queue: Limit=614.40 MB Total=0 Peak=116.12 KB
> E     Data Stream Manager Early RPCs: Total=0 Peak=6.76 KB
> E     TCMalloc Overhead: Total=103.56 MB
> E     RequestPool=fe-eval-exprs: Total=0 Peak=52.83 KB
> E       Query(d24ea53242b4cedc:fc8e088500000000): Total=0 Peak=0
> E     RequestPool=default-pool: Total=5.20 GB Peak=5.20 GB
> E       Query(f4014f7bb49ea78:6b926b1900000000): memory limit exceeded. 
> Limit=64.00 MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=1.76 GB 
> Total=1.76 GB Peak=1.96 GB
> E       Query(2c44b65fbcb4e1ce:3d73badf00000000): memory limit exceeded. 
> Limit=64.00 MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=1.04 GB 
> Total=1.04 GB Peak=1.61 GB
> E       Query(214cc23c1376176f:7844977b00000000): memory limit exceeded. 
> Limit=64.00 MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=1.23 GB 
> Total=1.23 GB Peak=1.23 GB
> E       Query(8949bdf792a32ad2:33a36c0300000000): memory limit exceeded. 
> Limit=64.00 MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=642.20 MB 
> Total=642.20 MB Peak=642.20 MB
> E       Query(5412ff4e6065721:519d3e6100000000): memory limit exceeded. 
> Limit=64.00 MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=556.49 MB 
> Total=556.49 MB Peak=556.49 MB
> E     Untracked Memory: Total=10.98 GB{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

Reply via email to