[
https://issues.apache.org/jira/browse/IMPALA-5746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17080083#comment-17080083
]
Sahil Takiar commented on IMPALA-5746:
--------------------------------------
Positing some ideas about how to fix this before I forget:
* Decrease the 10 minute timeout to a lower value
** Perhaps we should do this regardless, 10 minutes seems like a long time to
wait to detect a failure
** I guess the original intention of this flag is to handle the case where a
backend is still part of the cluster membership, but sending status reports is
failing for some other reason, so the timeout should be longer than the
statestore heartbeat timeout (I think this is ~10 seconds by default)
** Not sure what the exact value should be, but I'm going to throw out 1
minute as a suggestion - if a query can't report its status for over a minute,
then something is probably really wrong with that impalad
** Maybe we should integrate this with node blacklisting? I haven't looked at
this patch in detail, but it looks like we cancel all queries running on a
backend that is "unresponsive", but do we prevent future queries from running
on the fragment?
* Make executors listen to the cluster membership topic and have the
QueryExecMgr cancel all fragments running on a coordinator that has left the
cluster membership - I think we already do something similar inside the
Coordinator:
[https://github.com/apache/impala/blob/5989900ae81a98d6977bdd60f2281da47e9f69b7/be/src/runtime/exec-env.cc#L555]
> Remote fragments continue to hold onto memory after stopping the coordinator
> daemon
> -----------------------------------------------------------------------------------
>
> Key: IMPALA-5746
> URL: https://issues.apache.org/jira/browse/IMPALA-5746
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
> Affects Versions: Impala 2.10.0
> Reporter: Mostafa Mokhtar
> Assignee: Sahil Takiar
> Priority: Critical
> Attachments: remote_fragments_holding_memory.txt
>
>
> Repro
> # Start running queries
> # Kill the coordinator node
> # On the running Impalad check the memz tab, remote fragments continue to run
> and hold on to resources
> Remote fragments held on to memory +30 minutes after stopping the coordinator
> service.
> Attached thread dump from an Impalad running remote fragments .
> Snapshot of memz tab 30 minutes after killing the coordinator
> {code}
> Process: Limit=201.73 GB Total=5.32 GB Peak=179.36 GB
> Free Disk IO Buffers: Total=1.87 GB Peak=1.87 GB
> RequestPool=root.default: Total=1.35 GB Peak=178.51 GB
> Query(f64169d4bb3c901c:3a21d8ae00000000): Total=2.64 MB Peak=104.73 MB
> Fragment f64169d4bb3c901c:3a21d8ae00000051: Total=2.64 MB Peak=2.67 MB
> AGGREGATION_NODE (id=15): Total=2.54 MB Peak=2.57 MB
> Exprs: Total=30.12 KB Peak=30.12 KB
> EXCHANGE_NODE (id=14): Total=0 Peak=0
> DataStreamRecvr: Total=0 Peak=12.29 KB
> DataStreamSender (dst_id=17): Total=85.31 KB Peak=85.31 KB
> CodeGen: Total=1.53 KB Peak=374.50 KB
> Block Manager: Limit=161.39 GB Total=512.00 KB Peak=1.54 MB
> Query(2a4f12b3b4b1dc8c:db7e8cf200000000): Total=258.29 MB Peak=412.98 MB
> Fragment 2a4f12b3b4b1dc8c:db7e8cf20000008c: Total=2.29 MB Peak=2.29 MB
> SORT_NODE (id=11): Total=4.00 KB Peak=4.00 KB
> AGGREGATION_NODE (id=20): Total=2.27 MB Peak=2.27 MB
> Exprs: Total=25.12 KB Peak=25.12 KB
> EXCHANGE_NODE (id=19): Total=0 Peak=0
> DataStreamRecvr: Total=0 Peak=0
> DataStreamSender (dst_id=21): Total=3.88 KB Peak=3.88 KB
> CodeGen: Total=4.17 KB Peak=1.05 MB
> Block Manager: Limit=161.39 GB Total=256.25 MB Peak=321.66 MB
> Query(68421d2a5dea0775:83f5d97200000000): Total=282.77 MB Peak=443.53 MB
> Fragment 68421d2a5dea0775:83f5d9720000004a: Total=26.77 MB Peak=26.92 MB
> SORT_NODE (id=8): Total=8.00 KB Peak=8.00 KB
> Exprs: Total=4.00 KB Peak=4.00 KB
> ANALYTIC_EVAL_NODE (id=7): Total=4.00 KB Peak=4.00 KB
> Exprs: Total=4.00 KB Peak=4.00 KB
> SORT_NODE (id=6): Total=24.00 MB Peak=24.00 MB
> AGGREGATION_NODE (id=12): Total=2.72 MB Peak=2.83 MB
> Exprs: Total=85.12 KB Peak=85.12 KB
> EXCHANGE_NODE (id=11): Total=0 Peak=0
> DataStreamRecvr: Total=0 Peak=84.80 KB
> DataStreamSender (dst_id=13): Total=1.27 KB Peak=1.27 KB
> CodeGen: Total=24.80 KB Peak=4.13 MB
> Block Manager: Limit=161.39 GB Total=280.50 MB Peak=286.52 MB
> Query(e94c89fa89a74d27:82812bf900000000): Total=258.29 MB Peak=436.85 MB
> Fragment e94c89fa89a74d27:82812bf90000008e: Total=2.29 MB Peak=2.29 MB
> SORT_NODE (id=11): Total=4.00 KB Peak=4.00 KB
> AGGREGATION_NODE (id=20): Total=2.27 MB Peak=2.27 MB
> Exprs: Total=25.12 KB Peak=25.12 KB
> EXCHANGE_NODE (id=19): Total=0 Peak=0
> DataStreamRecvr: Total=0 Peak=0
> DataStreamSender (dst_id=21): Total=3.88 KB Peak=3.88 KB
> CodeGen: Total=4.17 KB Peak=1.05 MB
> Block Manager: Limit=161.39 GB Total=256.25 MB Peak=321.62 MB
> Query(4e43dad3bdc935d8:938b8b7e00000000): Total=2.65 MB Peak=105.60 MB
> Fragment 4e43dad3bdc935d8:938b8b7e00000052: Total=2.65 MB Peak=2.68 MB
> AGGREGATION_NODE (id=15): Total=2.55 MB Peak=2.57 MB
> Exprs: Total=30.12 KB Peak=30.12 KB
> EXCHANGE_NODE (id=14): Total=0 Peak=0
> DataStreamRecvr: Total=0 Peak=13.68 KB
> DataStreamSender (dst_id=17): Total=91.41 KB Peak=91.41 KB
> CodeGen: Total=1.53 KB Peak=374.50 KB
> Block Manager: Limit=161.39 GB Total=512.00 KB Peak=1.30 MB
> Query(b34bdd65f1ed017e:5a0291bd00000000): Total=2.37 MB Peak=106.56 MB
> Fragment b34bdd65f1ed017e:5a0291bd0000004b: Total=2.37 MB Peak=2.37 MB
> SORT_NODE (id=6): Total=4.00 KB Peak=4.00 KB
> AGGREGATION_NODE (id=10): Total=2.35 MB Peak=2.35 MB
> Exprs: Total=34.12 KB Peak=34.12 KB
> EXCHANGE_NODE (id=9): Total=0 Peak=0
> DataStreamRecvr: Total=0 Peak=4.23 KB
> DataStreamSender (dst_id=11): Total=3.45 KB Peak=3.45 KB
> CodeGen: Total=4.51 KB Peak=1.11 MB
> Block Manager: Limit=161.39 GB Total=256.00 KB Peak=912.81 KB
> Query(b74ba58d53b6c45f:3e8228600000000): Total=190.41 MB Peak=425.09 MB
> Fragment b74ba58d53b6c45f:3e822860000009f: Total=67.90 KB Peak=2.34 MB
> SORT_NODE (id=14): Total=4.00 KB Peak=4.00 KB
> HASH_JOIN_NODE (id=13): Total=42.25 KB Peak=42.25 KB
> Exprs: Total=9.12 KB Peak=9.12 KB
> Hash Join Builder (join_node_id=13): Total=9.12 KB Peak=9.12 KB
> Hash Join Builder (join_node_id=13) Exprs: Total=9.12 KB
> Peak=9.12 KB
> HDFS_SCAN_NODE (id=11): Total=0 Peak=0
> EXCHANGE_NODE (id=24): Total=0 Peak=0
> DataStreamRecvr: Total=0 Peak=0
> DataStreamSender (dst_id=25): Total=1.05 KB Peak=1.05 KB
> CodeGen: Total=12.59 KB Peak=2.29 MB
> Block Manager: Limit=161.39 GB Total=160.75 MB Peak=160.83 MB
> Fragment b74ba58d53b6c45f:3e8228600000085: Total=2.32 MB Peak=2.32 MB
> AGGREGATION_NODE (id=21): Total=2.29 MB Peak=2.29 MB
> Exprs: Total=44.12 KB Peak=44.12 KB
> EXCHANGE_NODE (id=20): Total=0 Peak=0
> DataStreamRecvr: Total=0 Peak=0
> DataStreamSender (dst_id=23): Total=22.09 KB Peak=22.09 KB
> CodeGen: Total=2.37 KB Peak=546.00 KB
> Fragment b74ba58d53b6c45f:3e8228600000060: Total=188.02 MB Peak=188.34
> MB
> Runtime Filter Bank: Total=16.00 MB Peak=16.00 MB
> AGGREGATION_NODE (id=9): Total=1.67 MB Peak=1.67 MB
> Exprs: Total=44.12 KB Peak=44.12 KB
> HASH_JOIN_NODE (id=8): Total=1.13 MB Peak=1.15 MB
> Exprs: Total=9.12 KB Peak=9.12 KB
> Hash Join Builder (join_node_id=8): Total=1.01 MB Peak=1.02 MB
> Hash Join Builder (join_node_id=8) Exprs: Total=9.12 KB Peak=9.12
> KB
> HASH_JOIN_NODE (id=7): Total=169.14 MB Peak=169.14 MB
> Exprs: Total=9.12 KB Peak=9.12 KB
> Hash Join Builder (join_node_id=7): Total=169.01 MB Peak=169.02 MB
> Hash Join Builder (join_node_id=7) Exprs: Total=9.12 KB Peak=9.12
> KB
> EXCHANGE_NODE (id=17): Total=0 Peak=0
> DataStreamRecvr: Total=0 Peak=587.50 KB
> EXCHANGE_NODE (id=18): Total=0 Peak=0
> DataStreamRecvr: Total=0 Peak=316.11 KB
> EXCHANGE_NODE (id=19): Total=0 Peak=0
> DataStreamRecvr: Total=0 Peak=4.70 KB
> DataStreamSender (dst_id=20): Total=58.39 KB Peak=58.39 KB
> CodeGen: Total=16.80 KB Peak=2.83 MB
> Query(cb4c14997ad6add2:c8f120100000000): Total=190.36 MB Peak=443.00 MB
> Fragment cb4c14997ad6add2:c8f1201000000a4: Total=67.90 KB Peak=2.34 MB
> SORT_NODE (id=14): Total=4.00 KB Peak=4.00 KB
> HASH_JOIN_NODE (id=13): Total=42.25 KB Peak=42.25 KB
> Exprs: Total=9.12 KB Peak=9.12 KB
> Hash Join Builder (join_node_id=13): Total=9.12 KB Peak=9.12 KB
> Hash Join Builder (join_node_id=13) Exprs: Total=9.12 KB
> Peak=9.12 KB
> HDFS_SCAN_NODE (id=11): Total=0 Peak=0
> EXCHANGE_NODE (id=24): Total=0 Peak=0
> DataStreamRecvr: Total=0 Peak=0
> DataStreamSender (dst_id=25): Total=1.05 KB Peak=1.05 KB
> CodeGen: Total=12.59 KB Peak=2.29 MB
> Block Manager: Limit=161.39 GB Total=160.75 MB Peak=160.83 MB
> Fragment cb4c14997ad6add2:c8f120100000088: Total=2.33 MB Peak=2.33 MB
> AGGREGATION_NODE (id=21): Total=2.29 MB Peak=2.29 MB
> Exprs: Total=44.12 KB Peak=44.12 KB
> EXCHANGE_NODE (id=20): Total=0 Peak=0
> DataStreamRecvr: Total=0 Peak=0
> DataStreamSender (dst_id=23): Total=26.83 KB Peak=26.83 KB
> CodeGen: Total=2.37 KB Peak=546.00 KB
> Fragment cb4c14997ad6add2:c8f120100000063: Total=187.97 MB Peak=188.08
> MB
> Runtime Filter Bank: Total=16.00 MB Peak=16.00 MB
> AGGREGATION_NODE (id=9): Total=1.67 MB Peak=1.67 MB
> Exprs: Total=44.12 KB Peak=44.12 KB
> HASH_JOIN_NODE (id=8): Total=1.14 MB Peak=1.15 MB
> Exprs: Total=9.12 KB Peak=9.12 KB
> Hash Join Builder (join_node_id=8): Total=1.01 MB Peak=1.02 MB
> Hash Join Builder (join_node_id=8) Exprs: Total=9.12 KB Peak=9.12
> KB
> HASH_JOIN_NODE (id=7): Total=169.07 MB Peak=169.14 MB
> Exprs: Total=9.12 KB Peak=9.12 KB
> Hash Join Builder (join_node_id=7): Total=169.01 MB Peak=169.02 MB
> Hash Join Builder (join_node_id=7) Exprs: Total=9.12 KB Peak=9.12
> KB
> EXCHANGE_NODE (id=17): Total=0 Peak=0
> DataStreamRecvr: Total=0 Peak=314.15 KB
> EXCHANGE_NODE (id=18): Total=0 Peak=0
> DataStreamRecvr: Total=0 Peak=861.18 KB
> EXCHANGE_NODE (id=19): Total=0 Peak=0
> DataStreamRecvr: Total=0 Peak=4.70 KB
> DataStreamSender (dst_id=20): Total=58.39 KB Peak=58.39 KB
> CodeGen: Total=16.80 KB Peak=2.83 MB
> Query(f04a57ce97102dd7:c2a1081700000000): Total=190.31 MB Peak=419.11 MB
> Fragment f04a57ce97102dd7:c2a1081700000085: Total=2.33 MB Peak=2.33 MB
> AGGREGATION_NODE (id=21): Total=2.29 MB Peak=2.29 MB
> Exprs: Total=44.12 KB Peak=44.12 KB
> EXCHANGE_NODE (id=20): Total=0 Peak=0
> DataStreamRecvr: Total=0 Peak=0
> DataStreamSender (dst_id=23): Total=23.67 KB Peak=23.67 KB
> CodeGen: Total=2.37 KB Peak=546.00 KB
> Block Manager: Limit=161.39 GB Total=160.75 MB Peak=160.83 MB
> Fragment f04a57ce97102dd7:c2a1081700000060: Total=187.99 MB Peak=188.07
> MB
> Runtime Filter Bank: Total=16.00 MB Peak=16.00 MB
> AGGREGATION_NODE (id=9): Total=1.68 MB Peak=1.68 MB
> Exprs: Total=44.12 KB Peak=44.12 KB
> HASH_JOIN_NODE (id=8): Total=1.14 MB Peak=1.15 MB
> Exprs: Total=9.12 KB Peak=9.12 KB
> Hash Join Builder (join_node_id=8): Total=1.01 MB Peak=1.02 MB
> Hash Join Builder (join_node_id=8) Exprs: Total=9.12 KB Peak=9.12
> KB
> HASH_JOIN_NODE (id=7): Total=169.09 MB Peak=169.14 MB
> Exprs: Total=9.12 KB Peak=9.12 KB
> Hash Join Builder (join_node_id=7): Total=169.01 MB Peak=169.02 MB
> Hash Join Builder (join_node_id=7) Exprs: Total=9.12 KB Peak=9.12
> KB
> EXCHANGE_NODE (id=17): Total=0 Peak=0
> DataStreamRecvr: Total=0 Peak=156.71 KB
> EXCHANGE_NODE (id=18): Total=0 Peak=0
> DataStreamRecvr: Total=0 Peak=1.32 MB
> EXCHANGE_NODE (id=19): Total=0 Peak=0
> DataStreamRecvr: Total=0 Peak=4.70 KB
> DataStreamSender (dst_id=20): Total=58.39 KB Peak=58.39 KB
> CodeGen: Total=16.80 KB Peak=2.83 MB
> Untracked Memory: Total=2.10 GB
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
