Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19251 )
Change subject: IMPALA-11730: Add support for spilling to Ozone ...................................................................... IMPALA-11730: Add support for spilling to Ozone Adds support for spilling to Ozone (ofs and o3fs schemes) for parity with HDFS. Note that ofs paths start with <volume>/<bucket>, which have naming restrictions; tmp/impala-scratch is a valid name, so something like ofs://localhost:9862/tmp would work as a scratch directory (volume tmp, implicit bucket impala-scratch). Updates tests to determine the correct path from the environment. Fixes backend tests to work with Ozone as well. Guards test_scratch_disk.py behind a new flag for filesystems that support spilling. Updates metric verification to wait for scratch-space-bytes-used to be non-zero, as it seems to update slower with Ozone. Refactors TmpDir to remove extraneous variables and functions. Each implementation is expected to handle its own token parsing. Initializes default_fs in ExecEnv when using TestEnv. Previously it was uninitialized, and uses of default_fs would return an empty string. Testing: - Ran backend, end-to-end, and custom cluster tests with Ozone. - Ran test_scratch_disk.py exhaustive runs with Ozone and HDFS. Change-Id: I5837c30357363f727ca832fb94169f2474fb4f6f Reviewed-on: http://gerrit.cloudera.org:8080/19251 Reviewed-by: Michael Smith <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/runtime/bufferpool/buffer-pool-test.cc M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/test-env.cc M be/src/runtime/test-env.h M be/src/runtime/tmp-file-mgr-internal.h M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M tests/common/skip.py M tests/custom_cluster/test_scratch_disk.py 9 files changed, 206 insertions(+), 226 deletions(-) Approvals: Michael Smith: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/19251 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5837c30357363f727ca832fb94169f2474fb4f6f Gerrit-Change-Number: 19251 Gerrit-PatchSet: 14 Gerrit-Owner: Michael Smith <[email protected]> Gerrit-Reviewer: Abhishek Rawat <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Yida Wu <[email protected]>
