Sahil Takiar created IMPALA-8523:
------------------------------------
Summary: Migrate hdfsOpen to builder-based openFile API
Key: IMPALA-8523
URL: https://issues.apache.org/jira/browse/IMPALA-8523
Project: IMPALA
Issue Type: Improvement
Components: Backend
Reporter: Sahil Takiar
Assignee: Sahil Takiar
When opening files via libhdfs we call {{hdfsOpen}} which ultimately calls
{{FileSystem#open(Path f, int bufferSize)}}. As of HADOOP-15229, the
HDFS-client now exposes a new API for opening files called {{openFile}}. The
new API has a few advantages (1) it is capable of specifying file specific
configuration values in a builder-based manner (see {{o.a.h.fs.FSBuilder}} for
details), and (2) it can open files asynchronously (e.g. see
{{o.a.h.fs.FutureDataInputStreamBuilder}} for details.
The async file opens are similar to IMPALA-7738 (Implement timeouts for HDFS
open calls). To avoid overlap between IMPALA-7738 and the async file opens in
{{openFile}}, HADOOP-15691 can be used to check which filesystems open files
asynchronously and which ones don't (currently only S3A opens files
asynchronously).
The main use case for the new {{openFile}} API is Impala-S3 performance.
Performance benchmarks have shown that setting
{{fs.s3a.experimental.input.fadvise}} to {{RANDOM}} for Parquet files can
significantly improve performance, however, this setting also adversely affects
scans of non-splittable file formats such as gzipped files (see HADOOP-13203).
One solution to this issue is to just document that setting
{{fs.s3a.experimental.input.fadvise}} to {{RANDOM}} for Parquet improves
performance, however, a better solution would be to use the new {{openFile}}
API to specify different values of fadvise depending on the file type.
This work is dependent on exposing the new {{openFile}} API via libhdfs
(HDFS-14478).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
