Hello Yida Wu, Abhishek Rawat, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19251
to look at the new patch set (#12).
Change subject: IMPALA-11730: Add support for spilling to Ozone
......................................................................
IMPALA-11730: Add support for spilling to Ozone
Adds support for spilling to Ozone (ofs and o3fs schemes) for parity
with HDFS. Note that ofs paths start with <volume>/<bucket>, which have
naming restrictions; tmp/impala-scratch is a valid name, so something
like ofs://localhost:9862/tmp would work as a scratch directory (volume
tmp, implicit bucket impala-scratch).
Updates tests to determine the correct path from the environment. Fixes
backend tests to work with Ozone as well. Guards test_scratch_disk.py
behind a new flag for filesystems that support spilling. Updates metric
verification to wait for scratch-space-bytes-used to be non-zero, as it
seems to update slower with Ozone.
Refactors TmpDir to remove extraneous variables and functions. Each
implementation is expected to handle its own token parsing.
Initializes default_fs in ExecEnv when using TestEnv. Previously it was
uninitialized, and uses of default_fs would return an empty string.
Testing:
- Ran backend, end-to-end, and custom cluster tests with Ozone.
- Ran test_scratch_disk.py exhaustive runs with Ozone and HDFS.
Change-Id: I5837c30357363f727ca832fb94169f2474fb4f6f
---
M be/src/runtime/bufferpool/buffer-pool-test.cc
M be/src/runtime/io/disk-io-mgr-test.cc
M be/src/runtime/test-env.cc
M be/src/runtime/test-env.h
M be/src/runtime/tmp-file-mgr-internal.h
M be/src/runtime/tmp-file-mgr-test.cc
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/hdfs-util.h
M tests/common/skip.py
M tests/custom_cluster/test_scratch_disk.py
10 files changed, 207 insertions(+), 226 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/19251/12
--
To view, visit http://gerrit.cloudera.org:8080/19251
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5837c30357363f727ca832fb94169f2474fb4f6f
Gerrit-Change-Number: 19251
Gerrit-PatchSet: 12
Gerrit-Owner: Michael Smith <[email protected]>
Gerrit-Reviewer: Abhishek Rawat <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Yida Wu <[email protected]>
