Daniel Vanko has uploaded this change for review. (
http://gerrit.cloudera.org:8080/24636
Change subject: IMPALA-8523: Migrate hdfsOpen to builder-based openFile API
......................................................................
IMPALA-8523: Migrate hdfsOpen to builder-based openFile API
Replace the deprecated hdfsOpenFile() on the scan read path with the
libhdfs builder-based openFile API (HADOOP-15229, exposed via libhdfs in
HDFS-14478). The builder lets Impala declare per-file configuration when
opening a file, which is the basis for the S3A read optimizations below.
OpenHdfsFileOp::Execute() now uses hdfsOpenFileBuilderAlloc/Opt/Build and
blocks on the returned future via hdfsOpenFileFutureGet() inside the
SynchronousThreadPool worker, so the existing open timeout (IMPALA-7738)
is preserved. Options are threaded from the scan node down to the open
call: HdfsFileDesc::GetFileInfo() computes them once per file so that all
scan ranges for the file (including footer/column ranges) inherit them
via ScanRange::GetFileInfo().
Unlike hdfsOpenFile(), the builder future (hdfsOpenFileFutureGet()) does not
translate the underlying Java exception to errno, so a missing file would
report "Unknown error" instead of ENOENT. OpenHdfsFileOp::Execute() detects a
file-not-found root cause and restores errno to ENOENT before building the
message, preserving the "No such file or directory" surface that scanners and
tests rely on.
The options are set with the standard, filesystem-agnostic openFile keys
and are all soft (hdfsOpenFileBuilderOpt), so filesystems that do not
understand a key ignore it and non-S3A reads are unaffected:
- fs.option.openfile.read.policy, chosen per file format. Each value is
an ordered "first recognized policy wins" list: the precise format
token first, for newer Hadoop runtimes, then a generic fallback that
is what current runtimes (which recognize only random/sequential/
adaptive) actually select. The axis is the access pattern, not
splittability:
Parquet, HUDI_PARQUET -> "parquet, random"
ORC -> "orc, random"
RC_FILE -> "columnar, random"
Avro -> "avro, sequential"
JSON -> "json, sequential"
text, SequenceFile -> "sequential"
Columnar formats read a footer then seek to selected column chunks
(random); row-oriented formats read front-to-back (sequential), so
forcing "random" on Avro/SequenceFile would hurt them. Kudu and JDBC
are not read through this path and get no policy. The format tokens
(parquet/orc/columnar) may enable S3A's analytics/prefetching stream
on a future Hadoop upgrade; that should be benchmarked against plain
"random" at that point rather than assumed to be a win.
- fs.option.openfile.length when the file length is known (> 0), so
S3A can skip the HEAD request on open. Guarded against a stale zero
length that would truncate reads.
No new flags are added: the read policy is a correctness improvement, and
soft options make the change safe across filesystems.
Testing:
- New be unit test OpenFileOptionsUtilTest for the pure option-building
logic (per-format policy mapping, S3A vs non-S3A, length guard).
- Parquet scanner e2e regression (TestScannersAllTableFormats) passes,
exercising the builder open path for real reads.
- Tests that asserted on the old hdfsOpenFile() error text were updated for
the API switch and re-run locally: test_hdfs_timeout (open-timeout message
is now "openFile builder for ... at backend ..."), and the missing-file
surface is preserved so test_stale_metadata::test_delete_file and
test_refresh_partition::test_remove_data_and_refresh still see
"Error(2): No such file or directory".
- Manual S3A validation on both local MinIO (server-side request trace)
and real AWS S3 (client-side S3A audit logging), comparing this commit
against its parent on an identical cold 24-file Parquet scan:
SELECT count(*), sum(int_col), max(string_col) FROM s3_alltypes
HEAD requests GET requests
parent (baseline) 24 72
this commit 0 72
The known-length hint eliminates the per-file HEAD on open (24 -> 0)
with no change to the read GET pattern (72 = 24 files x 3 ranges),
confirming the S3A runtime honors fs.option.openfile.length. The read
policy was verified at the HTTP layer: on a 69 MiB / 16-column file, a
single-column scan under "parquet, random" issues a tight ranged GET
for just that column (range[4-3497116], ~3.3 MiB) versus an open-ended
GET to EOF (range[4-72589876], ~69 MiB) under "sequential".
Change-Id: I46d810b19fe7d4859e3c2bcd7568b61fe73408c1
Assisted-by: Claude Opus 4.8 (Claude Code)
---
M be/src/exec/CMakeLists.txt
M be/src/exec/hdfs-scan-node-base.h
A be/src/exec/open-file-options-util-test.cc
A be/src/exec/open-file-options-util.cc
A be/src/exec/open-file-options-util.h
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/disk-io-mgr.h
M be/src/runtime/io/handle-cache.h
M be/src/runtime/io/handle-cache.inline.h
M be/src/runtime/io/hdfs-file-reader.cc
M be/src/runtime/io/hdfs-monitored-ops.cc
M be/src/runtime/io/hdfs-monitored-ops.h
A be/src/runtime/io/open-file-options.h
M be/src/runtime/io/request-ranges.h
M be/src/runtime/io/scan-range.cc
M tests/custom_cluster/test_hdfs_timeout.py
16 files changed, 451 insertions(+), 57 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/24636/3
--
To view, visit http://gerrit.cloudera.org:8080/24636
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I46d810b19fe7d4859e3c2bcd7568b61fe73408c1
Gerrit-Change-Number: 24636
Gerrit-PatchSet: 3
Gerrit-Owner: Daniel Vanko <[email protected]>
