Hello Tamas Mate, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14406
to look at the new patch set (#2).
Change subject: IMPALA-9033: log on slow HDFS I/Os
......................................................................
IMPALA-9033: log on slow HDFS I/Os
This logs a message with the time taken and also logs basic HDFS
statistics from the I/O, which would tell us if it was a remote
read, etc.
The threshold is 10s and is configurable via
--fs_slow_read_log_threshold_ms in case we want to make it more or less
sensitive.
Here's some example output that I obtained by adding a 500ms sleep
to the code path, and lowering the threshold to 500ms:
I1010 12:09:38.211959 30292 hdfs-file-reader.cc:173]
2448e3196bf9ee94:69adb16f00000001] Slow FS I/O operation on
hdfs://localhost:20500/test-warehouse/tpch.lineitem/lineitem.tbl for instance
2448e3196bf9ee94:69adb16f00000001 of query 2448e3196bf9ee94:69adb16f00000000.
Last read returned 8.00 MB. This thread has read 8.00 MB/8.00 MB starting at
offset 394264576 in this I/O scheduling quantum and taken 584.129ms so far. I/O
status: OK
I1010 12:09:38.212011 30292 hdfs-file-reader.cc:353]
2448e3196bf9ee94:69adb16f00000001] Stats for last read by this I/O thread:
totalBytesRead=8388608 totalLocalBytesRead=8388608
totalShortCircuitBytesRead=8388608 totalZeroCopyBytesRead=0
Change-Id: I1929921495706b482d91d91cffe27bee4478f5c4
---
M be/src/runtime/io/hdfs-file-reader.cc
M be/src/runtime/io/hdfs-file-reader.h
M be/src/util/runtime-profile-counters.h
3 files changed, 40 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/14406/2
--
To view, visit http://gerrit.cloudera.org:8080/14406
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1929921495706b482d91d91cffe27bee4478f5c4
Gerrit-Change-Number: 14406
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Tamas Mate <[email protected]>
