Matthew Jacobs created IMPALA-5790:
--------------------------------------
Summary: Failure to get reservation should print buffer pool limit
Key: IMPALA-5790
URL: https://issues.apache.org/jira/browse/IMPALA-5790
Project: IMPALA
Issue Type: Improvement
Components: Backend
Affects Versions: Impala 2.10.0
Reporter: Matthew Jacobs
Assignee: Matthew Jacobs
When playing around with a custom buffer pool limit, i.e.
{{--buffer_pool_limit}}, I found that failure to reserve enough memory error
message prints the mem tracker hierarchy but does not include the
buffer_pool_limit.
E.g.
{code}
start-impala-cluster.py --impalad_args="--buffer_pool_limit=2300m "
{code}
{code}
[localhost:21001] > set DEFAULT_SPILLABLE_BUFFER_SIZE=33554432;
DEFAULT_SPILLABLE_BUFFER_SIZE set to 33554432
[localhost:21001] > set MIN_SPILLABLE_BUFFER_SIZE=33554432;
MIN_SPILLABLE_BUFFER_SIZE set to 33554432
[localhost:21001] > select straight_join * from tpch_parquet.lineitem join
(select n_regionkey, count(distinct n_nationkey) from tpch_parquet.nation group
by 1) v on n_regionkey = l_orderkey;
Query: select straight_join * from tpch_parquet.lineitem join (select
n_regionkey, count(distinct n_nationkey) from tpch_parquet.nation group by 1) v
on n_regionkey = l_orderkey
Query submitted at: 2017-08-10 18:54:14 (Coordinator:
http://mj-desktop.ca.cloudera.com:25001)
ERROR: ExecQueryFInstances rpc query_id=b94a87085426982d:641c5a9600000000
failed: Failed to get minimum memory reservation of 1.59 GB on daemon
mj-desktop.ca.cloudera.com:22000 for query b94a87085426982d:641c5a9600000000
because it would exceed an applicable query, request pool or process memory
limit. Memory usage:
Process: Limit=8.35 GB Total=1.11 GB Peak=1.94 GB
Buffer Pool: Free Buffers: Total=768.00 MB
Buffer Pool: Clean Pages: Total=0
Free Disk IO Buffers: Total=256.00 KB Peak=68.00 MB
RequestPool=default-pool: Total=621.45 MB Peak=1.75 GB
Query(2540ad2fb916785d:17301f8a00000000): Reservation=544.00 MB
ReservationLimit=6.68 GB OtherMemory=77.45 MB Total=621.45 MB Peak=1.75 GB
Fragment 2540ad2fb916785d:17301f8a00000000: Reservation=0
OtherMemory=16.00 KB Total=16.00 KB Peak=16.00 KB
EXCHANGE_NODE (id=8): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
PLAN_ROOT_SINK: Total=0 Peak=0
CodeGen: Total=0 Peak=0
Fragment 2540ad2fb916785d:17301f8a00000005: Reservation=544.00 MB
OtherMemory=77.43 MB Total=621.43 MB Peak=621.43 MB
Runtime Filter Bank: Total=1.00 MB Peak=1.00 MB
HASH_JOIN_NODE (id=4): Reservation=544.00 MB OtherMemory=42.25 KB
Total=544.04 MB Peak=544.06 MB
Exprs: Total=13.12 KB Peak=13.12 KB
Hash Join Builder (join_node_id=4): Total=13.12 KB Peak=21.12 KB
Hash Join Builder (join_node_id=4) Exprs: Total=13.12 KB Peak=13.12
KB
HDFS_SCAN_NODE (id=0): Total=76.35 MB Peak=76.35 MB
EXCHANGE_NODE (id=7): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=20.04 KB
DataStreamSender (dst_id=8): Total=1.27 KB Peak=1.27 KB
CodeGen: Total=21.40 KB Peak=1.41 MB
Fragment 2540ad2fb916785d:17301f8a00000002: Reservation=0 OtherMemory=0
Total=0 Peak=1.06 GB
AGGREGATION_NODE (id=3): Reservation=0 OtherMemory=0 Total=0
Peak=544.02 MB
AGGREGATION_NODE (id=6): Reservation=0 OtherMemory=0 Total=0
Peak=544.04 MB
EXCHANGE_NODE (id=5): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=20.20 KB
DataStreamSender (dst_id=7): Total=0 Peak=34.90 KB
CodeGen: Total=0 Peak=529.00 KB
Fragment 2540ad2fb916785d:17301f8a00000001: Reservation=0 OtherMemory=0
Total=0 Peak=160.21 MB
AGGREGATION_NODE (id=2): Reservation=0 OtherMemory=0 Total=0
Peak=160.02 MB
HDFS_SCAN_NODE (id=1): Total=0 Peak=157.00 KB
DataStreamSender (dst_id=5): Total=0 Peak=25.59 KB
CodeGen: Total=0 Peak=417.50 KB
Query(b94a87085426982d:641c5a9600000000): Reservation=0
ReservationLimit=6.68 GB OtherMemory=0 Total=0 Peak=0
Untracked Memory: Total=-266538560.00 B
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
