Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14406
Change subject: IMPALA-9033: log on slow HDFS I/Os ...................................................................... IMPALA-9033: log on slow HDFS I/Os This logs a message with the time taken and also logs basic HDFS statistics from the I/O, which would tell us if it was a remote read, etc. The threshold is 10s and is configurable via --fs_slow_read_log_threshold_ms in case we want to make it more or less sensitive. Here's some example output that I obtained by adding a 500ms sleep to the code path, and lowering the threshold to 500ms: I1009 16:12:29.732028 20517 hdfs-file-reader.cc:183] d3455c8d068b7833:0a731c1d00000003] Slow FS I/O operation on hdfs://localhost:20500/test-warehouse/tpch.lineitem/lineitem.tbl for instance d3455c8d068b7833:0a731c1d00000003 of query d3455c8d068b7833:0a731c1d00000000. Last read returned 8.00 MB. This thread has read 8.00 MB/8.00 MB starting at offset 0 in this I/O scheduling quantum. I1009 16:12:29.732102 20517 hdfs-file-reader.cc:348] d3455c8d068b7833:0a731c1d00000003] Stats for last read by this I/O thread: totalBytesRead=8388608 totalLocalBytesRead=8388608 totalShortCircuitBytesRead=8388608 totalZeroCopyBytesRead=0 Change-Id: I1929921495706b482d91d91cffe27bee4478f5c4 --- M be/src/runtime/io/hdfs-file-reader.cc M be/src/runtime/io/hdfs-file-reader.h M be/src/util/runtime-profile-counters.h 3 files changed, 36 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/14406/1 -- To view, visit http://gerrit.cloudera.org:8080/14406 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I1929921495706b482d91d91cffe27bee4478f5c4 Gerrit-Change-Number: 14406 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong <[email protected]>
