Sahil Takiar created IMPALA-8237:
------------------------------------
Summary: Enabling preads always fetches hedged reads metrics
Key: IMPALA-8237
URL: https://issues.apache.org/jira/browse/IMPALA-8237
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Sahil Takiar
Assignee: Sahil Takiar
In {{HdfsFileReader}} if preads are enabled, we assume that hedged reads are
enabled as well, so whenever we close a file we make a libhdfs call to collect
a few hedged read metrics from the underlying {{FileSystem}} object. However,
as part of IMPALA-5212 we may want to enable preads even when hedged reads are
disabled, so making the call to libhdfs to fetch hedged read metrics will be a
waste.
Digging through the HDFS code, it seems the HDFS client triggers hedged reads
only if {{dfs.client.hedged.read.threadpool.size}} is greater than 0. We can
use the same check in {{HdfsFileReader}} to trigger the fetch of hedged read
metrics. The issue is that currently libhdfs does not provide a good way of
getting the value of {{dfs.client.hedged.read.threadpool.size}}, it provides a
method called {{hdfsConfGetInt}}, but that method simply calls {{new
Configuration()}} and fetches the value of
{{dfs.client.hedged.read.threadpool.size}} from it. The issue is that calling
{{new Configuration}} simply loads the current {{hdfs-site.xml}},
{{core-site.xml}}, etc. which does not take into account the scenario where the
default configuration has been modified for specific filesystem objects - e.g.
using {{hdfsBuilder}} to set non-default configuration parameters (see
HDFS-14301 for more details).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
