IMPALA-4566: Kudu client glog contention can cause timeouts Under stressful workloads, there appears to be significant contention in glog resulting from Kudu logging, causing timeouts and failed queries.
An easy solution for now is to downgrade Kudu WARNINGs to INFOs as WARNINGs grab a lock to flush the log. This is appropriate as Kudu logs WARNINGs much more frequently than Impala and for things that Impala would normally consider INFO-level. Testing: Manually verified that the patch redirects Kudu WARNINGs to the INFO log. Not tested under stress to verify if this actually solves the contention problem. Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2 Reviewed-on: http://gerrit.cloudera.org:8080/5334 Reviewed-by: Matthew Jacobs <[email protected]> Tested-by: Internal Jenkins Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/2ae93b03 Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/2ae93b03 Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/2ae93b03 Branch: refs/heads/hadoop-next Commit: 2ae93b03ee00aff84e25db51813c9d1f5cb26b8b Parents: 02fc53d Author: Thomas Tauber-Marshall <[email protected]> Authored: Fri Dec 2 15:15:55 2016 -0800 Committer: Internal Jenkins <[email protected]> Committed: Fri Dec 9 23:28:41 2016 +0000 ---------------------------------------------------------------------- be/src/exec/kudu-util.cc | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/2ae93b03/be/src/exec/kudu-util.cc ---------------------------------------------------------------------- diff --git a/be/src/exec/kudu-util.cc b/be/src/exec/kudu-util.cc index 5afa0a4..e81baf5 100644 --- a/be/src/exec/kudu-util.cc +++ b/be/src/exec/kudu-util.cc @@ -78,14 +78,16 @@ void LogKuduMessage(void* unused, kudu::client::KuduLogSeverity severity, const char* filename, int line_number, const struct ::tm* time, const char* message, size_t message_len) { - // Note: we use raw ints instead of the nice LogSeverity typedef + // Note: use raw ints instead of the nice LogSeverity typedef // that can be found in glog/log_severity.h as it has an import // conflict with gutil/logging-inl.h (indirectly imported). int glog_severity; switch (severity) { case kudu::client::SEVERITY_INFO: glog_severity = 0; break; - case kudu::client::SEVERITY_WARNING: glog_severity = 1; break; + // Log Kudu WARNING messages at the INFO level to avoid contention created by glog + // locking while flushing WARNING messages. + case kudu::client::SEVERITY_WARNING: glog_severity = 0; break; case kudu::client::SEVERITY_ERROR: glog_severity = 2; break; case kudu::client::SEVERITY_FATAL: glog_severity = 3; break; default : DCHECK(false) << "Unexpected severity type: " << severity;
