Hello Yida Wu, Abhishek Rawat, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19251
to look at the new patch set (#9).
Change subject: IMPALA-11730: Add support for spilling to Ozone
......................................................................
IMPALA-11730: Add support for spilling to Ozone
Adds support for spilling to Ozone (ofs and o3fs schemes) for parity
with HDFS. Note that ofs paths start with <volume>/<bucket>, which have
naming restrictions; tmp/impala-scratch is a valid name, so something
like ofs://localhost:9862/tmp would work as a scratch directory (volume
tmp, implicit bucket impala-scratch).
Fixes backend tests to work with Ozone.
Updates tests to determine the correct path from the environment. Fixes
backend tests to work with Ozone as well. Guards test_scratch_disk.py
behind a new flag for filesystems that support spilling. Updates metric
verification to wait for scratch-space-bytes-used to be non-zero, as it
seems to update slower with Ozone.
Refactors TmpDir to remove extraneous variables and functions. Each
implementation is expected to handle its own token parsing.
Testing:
- Ran backend, end-to-end, and custom cluster tests with Ozone.
- Ran test_scratch_disk.py exhaustive runs with Ozone and HDFS.
Change-Id: I5837c30357363f727ca832fb94169f2474fb4f6f
---
M be/src/runtime/bufferpool/buffer-pool-test.cc
M be/src/runtime/io/disk-io-mgr-test.cc
M be/src/runtime/tmp-file-mgr-internal.h
M be/src/runtime/tmp-file-mgr-test.cc
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/hdfs-util.h
M tests/common/skip.py
M tests/custom_cluster/test_scratch_disk.py
8 files changed, 186 insertions(+), 197 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/19251/9
--
To view, visit http://gerrit.cloudera.org:8080/19251
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5837c30357363f727ca832fb94169f2474fb4f6f
Gerrit-Change-Number: 19251
Gerrit-PatchSet: 9
Gerrit-Owner: Michael Smith <[email protected]>
Gerrit-Reviewer: Abhishek Rawat <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Yida Wu <[email protected]>
