IMPALA-3517: S3: Get rid of hdfsDelete() warning message In the patch for IMPALA-3452, a bug was introduced which does an unnecessary call to hdfsDelete() on the staging directory of a partition(s), when the user selects the option of skipping the staging step. Also, this delete will not happen when it should happen, e.g on HDFS partitions. (Although it will be deleted by the coordinator later).
This patch fixes this bug by making sure that we delete the staging directory when it is necessary and skip the call to delete when we do not want it. Change-Id: I0a81ba0abfc24ee56689211579f46ac353e98adb Reviewed-on: http://gerrit.cloudera.org:8080/3019 Reviewed-by: Sailesh Mukil <[email protected]> Tested-by: Internal Jenkins Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/da49a37e Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/da49a37e Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/da49a37e Branch: refs/heads/master Commit: da49a37edc136bd97a7255a57325ede8d37e6b5e Parents: 5c56ec0 Author: Sailesh Mukil <[email protected]> Authored: Tue May 10 15:27:23 2016 -0700 Committer: Tim Armstrong <[email protected]> Committed: Thu May 12 14:18:03 2016 -0700 ---------------------------------------------------------------------- be/src/exec/hdfs-table-sink.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/da49a37e/be/src/exec/hdfs-table-sink.cc ---------------------------------------------------------------------- diff --git a/be/src/exec/hdfs-table-sink.cc b/be/src/exec/hdfs-table-sink.cc index d3a2ade..0a935eb 100644 --- a/be/src/exec/hdfs-table-sink.cc +++ b/be/src/exec/hdfs-table-sink.cc @@ -506,7 +506,7 @@ inline Status HdfsTableSink::GetOutputPartition(RuntimeState* state, state->per_partition_status()->insert( make_pair(partition->partition_name, partition_status)); - if (!no_more_rows && ShouldSkipStaging(state, partition)) { + if (!no_more_rows && !ShouldSkipStaging(state, partition)) { // Indicate that temporary directory is to be deleted after execution (*state->hdfs_files_to_move())[partition->tmp_hdfs_dir_name] = ""; }
