[ https://issues.apache.org/jira/browse/IMPALA-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong updated IMPALA-3544: ---------------------------------- Summary: crash in DiskIoMgr::GetFreeBuffer (was: data-stream-sender.cc:261 channel send status: Sender timed out waiting for receiver fragment instance: a045d3abcd1bed4f:a116563837c8ebda) > crash in DiskIoMgr::GetFreeBuffer > --------------------------------- > > Key: IMPALA-3544 > URL: https://issues.apache.org/jira/browse/IMPALA-3544 > Project: IMPALA > Issue Type: Bug > Components: Backend > Affects Versions: Impala 2.5.0 > Environment: Ubuntu Trusty, CDH 5.7.0 > Reporter: Scott Wallace > Assignee: Sailesh Mukil > Priority: Critical > Labels: crash, hang > Attachments: 2016-05-17_1441.png, 2016-05-18_1109.png, > 2016-05-18_1140.png, 2016-05-26_1438.png, 2016-05-26_1440.png, Screen Shot > 2016-05-27 at 12.43.23 PM.png, hs_err_pid7016.log, > impalad.ux-reporting-engine-worker-13-prod-us-east-1a.impala.log.ERROR.20160516-152100.89097, > > impalad.ux-reporting-engine-worker-18-prod-us-east-1a.impala.log.ERROR.20160519-140954.7016, > > impalad.ux-reporting-engine-worker-6-prod-us-east-1a.impala.log.INFO.20160516-152100.44312 > > > Impala becomes unresponsive, causing a full outage for our reporting system. > Impalad logs show various errors as follows: > {noformat} > Time Log Level Source Log Message > May 16, 3:20:59.953 PM ERROR logging.cc:120 > stderr will be logged to this file. > May 16, 3:27:38.652 PM ERROR data-stream-sender.cc:261 > channel send status: > Sender timed out waiting for receiver fragment instance: > a045d3abcd1bed4f:a116563837c8ebda > May 16, 3:27:39.738 PM ERROR data-stream-sender.cc:261 > channel send status: Couldn't open transport for > ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: > Connection timed out) > May 16, 3:27:44.784 PM ERROR data-stream-sender.cc:261 > channel send status: Couldn't open transport for > ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: > Connection timed out) > May 16, 3:27:46.055 PM ERROR data-stream-sender.cc:261 > channel send status: Couldn't open transport for > ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: > Connection timed out) > May 16, 3:27:46.371 PM ERROR data-stream-sender.cc:261 > channel send status: Couldn't open transport for > ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: > Connection timed out) > May 16, 3:27:46.906 PM ERROR data-stream-sender.cc:261 > channel send status: Couldn't open transport for > ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: > Connection timed out) > May 16, 3:27:51.003 PM ERROR data-stream-sender.cc:261 > channel send status: Couldn't open transport for > ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: > Connection timed out) > May 16, 3:27:52.479 PM ERROR data-stream-sender.cc:261 > channel send status: Couldn't open transport for > ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed: > Connection timed out) > May 16, 3:27:54.227 PM ERROR data-stream-sender.cc:261 > channel send status: Couldn't open transport for > ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: > Connection timed out) > May 16, 3:27:54.298 PM ERROR data-stream-sender.cc:261 > channel send status: Couldn't open transport for > ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed: > Connection timed out) > May 16, 3:27:54.305 PM ERROR data-stream-sender.cc:261 > channel send status: > Sender timed out waiting for receiver fragment instance: > 7446d34d9eefb9f:436f61bcd3c78eb3 > May 16, 3:27:55.022 PM ERROR data-stream-sender.cc:261 > channel send status: > Sender timed out waiting for receiver fragment instance: > b44445bcb782b5a4:5cef13b26af20aa1 > May 16, 3:27:55.646 PM ERROR data-stream-sender.cc:261 > channel send status: > Sender timed out waiting for receiver fragment instance: > 3249c6494703ecc7:3b9186fbed00dc > May 16, 3:27:56.042 PM ERROR data-stream-sender.cc:261 > channel send status: Couldn't open transport for > ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: > Connection timed out) > May 16, 3:27:56.277 PM ERROR data-stream-sender.cc:261 > channel send status: Couldn't open transport for > ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed: > Connection timed out) > May 16, 3:27:56.749 PM ERROR data-stream-sender.cc:261 > channel send status: Couldn't open transport for > ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: > Connection timed out) > May 16, 3:27:57.126 PM ERROR data-stream-sender.cc:261 > channel send status: Couldn't open transport for > ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed: > Connection timed out) > May 16, 3:27:59.030 PM ERROR data-stream-sender.cc:261 > channel send status: Couldn't open transport for > ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed: > Connection timed out) > May 16, 3:28:03.573 PM ERROR data-stream-sender.cc:261 > channel send status: > Sender timed out waiting for receiver fragment instance: > c64b22033608d609:10e070d34aa299dd > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org