Internal Jenkins has submitted this change and it was merged. Change subject: IMPALA-3106: Don't allow scanner threads to spin before ranges are issued ......................................................................
IMPALA-3106: Don't allow scanner threads to spin before ranges are issued Scanner threads are started in HdfsScanNode::Open(), before the actual scan ranges are issued in GetNext(). If there is a delay between these methods, the scanner threads will spin on DiskIoMgr::GetNextRange() which returns immediately thinking that all the scan ranges are finished (rather than not yet issued). This patch works around the issue by forcing scanner threads to wait until the initial ranges are issued. That wait is limited to 20ms, after which the thread will wake up to check if it should yield based on contention for thread tokens; otherwise it waits again. Fixing this drops CPU usage to nearly 0 during a large delay between GetNext() and Open() (e.g. by artificially delaying runtime filter delivery), where before nearly one CPU per scanner thread was consumed. Change-Id: I064de1ae037b578c70d65503895a920461af877a Reviewed-on: http://gerrit.cloudera.org:8080/2379 Reviewed-by: Marcel Kornacker <[email protected]> Tested-by: Internal Jenkins --- M be/src/exec/hdfs-scan-node.cc M be/src/exec/hdfs-scan-node.h M be/src/util/counting-barrier.h 3 files changed, 32 insertions(+), 4 deletions(-) Approvals: Marcel Kornacker: Looks good to me, approved Internal Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/2379 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I064de1ae037b578c70d65503895a920461af877a Gerrit-PatchSet: 4 Gerrit-Project: Impala Gerrit-Branch: cdh5-2.5.0_5.7.0 Gerrit-Owner: Henry Robinson <[email protected]> Gerrit-Reviewer: Henry Robinson <[email protected]> Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Marcel Kornacker <[email protected]> Gerrit-Reviewer: Skye Wanderman-Milne <[email protected]>
