[
https://issues.apache.org/jira/browse/IMPALA-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-3544:
----------------------------------
Summary: crash in DiskIoMgr::GetFreeBuffer (was: data-stream-sender.cc:261
channel send status: Sender timed out waiting for receiver fragment instance:
a045d3abcd1bed4f:a116563837c8ebda)
> crash in DiskIoMgr::GetFreeBuffer
> ---------------------------------
>
> Key: IMPALA-3544
> URL: https://issues.apache.org/jira/browse/IMPALA-3544
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.5.0
> Environment: Ubuntu Trusty, CDH 5.7.0
> Reporter: Scott Wallace
> Assignee: Sailesh Mukil
> Priority: Critical
> Labels: crash, hang
> Attachments: 2016-05-17_1441.png, 2016-05-18_1109.png,
> 2016-05-18_1140.png, 2016-05-26_1438.png, 2016-05-26_1440.png, Screen Shot
> 2016-05-27 at 12.43.23 PM.png, hs_err_pid7016.log,
> impalad.ux-reporting-engine-worker-13-prod-us-east-1a.impala.log.ERROR.20160516-152100.89097,
>
> impalad.ux-reporting-engine-worker-18-prod-us-east-1a.impala.log.ERROR.20160519-140954.7016,
>
> impalad.ux-reporting-engine-worker-6-prod-us-east-1a.impala.log.INFO.20160516-152100.44312
>
>
> Impala becomes unresponsive, causing a full outage for our reporting system.
> Impalad logs show various errors as follows:
> {noformat}
> Time Log Level Source Log Message
> May 16, 3:20:59.953 PM ERROR logging.cc:120
> stderr will be logged to this file.
> May 16, 3:27:38.652 PM ERROR data-stream-sender.cc:261
> channel send status:
> Sender timed out waiting for receiver fragment instance:
> a045d3abcd1bed4f:a116563837c8ebda
> May 16, 3:27:39.738 PM ERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:44.784 PM ERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:46.055 PM ERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:46.371 PM ERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:46.906 PM ERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:51.003 PM ERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:52.479 PM ERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:54.227 PM ERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:54.298 PM ERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:54.305 PM ERROR data-stream-sender.cc:261
> channel send status:
> Sender timed out waiting for receiver fragment instance:
> 7446d34d9eefb9f:436f61bcd3c78eb3
> May 16, 3:27:55.022 PM ERROR data-stream-sender.cc:261
> channel send status:
> Sender timed out waiting for receiver fragment instance:
> b44445bcb782b5a4:5cef13b26af20aa1
> May 16, 3:27:55.646 PM ERROR data-stream-sender.cc:261
> channel send status:
> Sender timed out waiting for receiver fragment instance:
> 3249c6494703ecc7:3b9186fbed00dc
> May 16, 3:27:56.042 PM ERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:56.277 PM ERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:56.749 PM ERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:57.126 PM ERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:59.030 PM ERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:28:03.573 PM ERROR data-stream-sender.cc:261
> channel send status:
> Sender timed out waiting for receiver fragment instance:
> c64b22033608d609:10e070d34aa299dd
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
