Repository: impala
Updated Branches:
  refs/heads/master ff3ddb51a -> ad7cbc94e


IMPALA-6549: Enable file handle cache by default

The file handle cache was disabled by default
due to two HDFS issues: HDFS-12528 and HDFS-14872.
Both have been fixed and the CDH components in
the toolchain include both fixes.

This reenables the file handle cache by default.

Change-Id: I6935825a1c4c7b2da0bb877f732027be1a57a8b7
Reviewed-on: http://gerrit.cloudera.org:8080/9371
Reviewed-by: Joe McDonnell <joemcdonn...@cloudera.com>
Tested-by: Impala Public Jenkins


Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/ad7cbc94
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/ad7cbc94
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/ad7cbc94

Branch: refs/heads/master
Commit: ad7cbc94e2099e0dd8f0d958602c7c46f196a72a
Parents: ff3ddb5
Author: Joe McDonnell <joemcdonn...@cloudera.com>
Authored: Tue Feb 20 16:37:29 2018 -0800
Committer: Impala Public Jenkins <impala-public-jenk...@gerrit.cloudera.org>
Committed: Fri Feb 23 05:02:31 2018 +0000

----------------------------------------------------------------------
 be/src/runtime/io/disk-io-mgr.cc | 16 +++++++---------
 1 file changed, 7 insertions(+), 9 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/impala/blob/ad7cbc94/be/src/runtime/io/disk-io-mgr.cc
----------------------------------------------------------------------
diff --git a/be/src/runtime/io/disk-io-mgr.cc b/be/src/runtime/io/disk-io-mgr.cc
index 6c7b9e6..0ac3669 100644
--- a/be/src/runtime/io/disk-io-mgr.cc
+++ b/be/src/runtime/io/disk-io-mgr.cc
@@ -98,10 +98,7 @@ DEFINE_int32(max_free_io_buffers, 128,
 // uses about 6kB of memory. 20k file handles will thus reserve ~120MB of 
memory.
 // The actual amount of memory that is associated with a file handle can be 
larger
 // or smaller, depending on the replication factor for this file or the path 
name.
-// TODO: This is currently disabled due to HDFS-12528, which can disable short 
circuit
-// reads when file handle caching is enabled. This should be reenabled by 
default
-// when that issue is fixed.
-DEFINE_uint64(max_cached_file_handles, 0, "Maximum number of HDFS file handles 
"
+DEFINE_uint64(max_cached_file_handles, 20000, "Maximum number of HDFS file 
handles "
     "that will be cached. Disabled if set to 0.");
 
 // The unused file handle timeout specifies how long a file handle will remain 
in the
@@ -112,11 +109,12 @@ DEFINE_uint64(max_cached_file_handles, 0, "Maximum number 
of HDFS file handles "
 // If a file is deleted through HDFS, this open file descriptor can keep the 
disk space
 // from being freed. When the metadata sees that a file has been deleted, the 
file handle
 // will no longer be used by future queries. Aging out this file handle allows 
the
-// disk space to be freed in an appropriate period of time.
-// TODO: HDFS-12528 (which can disable short circuit reads) is more likely to 
happen
-// if file handles are cached for longer than 5 minutes. Use a conservative 
value for
-// the unused file handle cache timeout until HDFS-12528 is fixed.
-DEFINE_uint64(unused_file_handle_timeout_sec, 270, "Maximum time, in seconds, 
that an "
+// disk space to be freed in an appropriate period of time. The default value 
is
+// 6 hours. This was chosen to be less than a typical value for HDFS's 
fs.trash.interval.
+// This means that when files are deleted via the trash, the file handle cache 
will
+// have evicted the file handle before the files are flushed from the trash. 
This
+// means that the file handle cache won't impact available disk space.
+DEFINE_uint64(unused_file_handle_timeout_sec, 21600, "Maximum time, in 
seconds, that an "
     "unused HDFS file handle will remain in the file handle cache. Disabled if 
set "
     "to 0.");
 

Reply via email to