[
https://issues.apache.org/jira/browse/IMPALA-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16762367#comment-16762367
]
ASF subversion and git services commented on IMPALA-7265:
---------------------------------------------------------
Commit 255ec4687ebe6195b20e5566394f3692c07e3b7f in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=255ec46 ]
IMPALA-7265: Enable caching of remote file handles by default
This changes the default value of cache_remote_file_handles
from false to true. Testing shows that this setting has a
major impact on performance for clusters that do remote HDFS
reads. Hand testing of the cache did not reveal any problems
with the semantics of caching remote file handles.
Change-Id: I2fc4a69c6bf721017f4adcdc302db9eace5135a4
Reviewed-on: http://gerrit.cloudera.org:8080/12387
Reviewed-by: Philip Zeyliger <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Cache remote file handles
> -------------------------
>
> Key: IMPALA-7265
> URL: https://issues.apache.org/jira/browse/IMPALA-7265
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Affects Versions: Impala 3.1.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Critical
>
> The file handle cache currently does not allow caching remote file handles.
> This means that clusters that have a lot of remote reads can suffer from
> overloading the NameNode. Impala should be able to cache remote file handles.
> There are some open questions about remote file handles and whether they
> behave differently from local file handles. In particular:
> # Is there any resource constraint on the number of remote file handles
> open? (e.g. do they maintain a network connection?)
> # Are there any semantic differences in how remote file handles behave when
> files are deleted, overwritten, or appended?
> # Are there any extra failure cases for remote file handles? (i.e. if a
> machine goes down or a remote file handle is left open for an extended period
> of time)
> The form of caching will depend on the answers, but at the very least, it
> should be possible to cache a remote file handle at the level of a query so
> that a Parquet file with multiple columns can share file handles.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
