Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17901 )
Change subject: IMPALA-10945: Fix S3 scratch path behavior ...................................................................... IMPALA-10945: Fix S3 scratch path behavior IMPALA-10429 "Support Spill to HDFS" introduces a new behavior to S3 scratch path. It added a path verification for the S3 path, however, the HdfsFsCache::GetNameNodeFromPath in the verification forces the input to have at least a directory after the authority, like "s3a://authority/dir", otherwise it will return an error and lead to a failure on the TmpFileMgr initialization. Therefore, it changes the previous behavior which was able to support "s3a://authority", and may affect current users. This patch resumes the behavior of the s3 scratch path to allow "s3a://authority". The solution is to pass the path of the user's input combined with a scratch suffix "/impala-scratch" to the verification function, therefore, at least one directory is contained in the path. Tests: Ran core tests. Added logic to run two types of path format in TmpFileMgrTest: "s3a://authority" and "s3a://authority/dir". Change-Id: I028f375b9f535f8641261cc02f921497e076aa9b Reviewed-on: http://gerrit.cloudera.org:8080/17901 Reviewed-by: Abhishek Rawat <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc 2 files changed, 85 insertions(+), 78 deletions(-) Approvals: Abhishek Rawat: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/17901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I028f375b9f535f8641261cc02f921497e076aa9b Gerrit-Change-Number: 17901 Gerrit-PatchSet: 3 Gerrit-Owner: Yida Wu <[email protected]> Gerrit-Reviewer: Abhishek Rawat <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
