This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 961ccf11012 [Fix](hive-writer) Fix s3 file commiter not working.
(#35502)
961ccf11012 is described below
commit 961ccf1101261f736516a1305e6044c3de15ebd8
Author: Qi Chen <[email protected]>
AuthorDate: Wed May 29 12:11:05 2024 +0800
[Fix](hive-writer) Fix s3 file commiter not working. (#35502)
### Issue
`The specified upload does not exist. The upload ID may be invalid, or
the upload may have been aborted or completed. (Service: S3, Status
Code: 404, Request ID: 66557027E897233333FFC198)`
### Root cause
#35311 adjusted the order of building hive partition update information.
This change caused the update id to be unable to be obtained, causing
the s3 file committer to not work properly.
### Solution
Because uploading to s3 occurs after s3 file writer close(), close()
must be called first, and then the build hive partiton update
information is called.
---
be/src/vec/sink/writer/vhive_partition_writer.cpp | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/be/src/vec/sink/writer/vhive_partition_writer.cpp
b/be/src/vec/sink/writer/vhive_partition_writer.cpp
index 10140c7a06d..f65eb0d0972 100644
--- a/be/src/vec/sink/writer/vhive_partition_writer.cpp
+++ b/be/src/vec/sink/writer/vhive_partition_writer.cpp
@@ -139,9 +139,6 @@ Status VHivePartitionWriter::open(RuntimeState* state,
RuntimeProfile* profile)
}
Status VHivePartitionWriter::close(const Status& status) {
- if (status.ok()) {
-
_state->hive_partition_updates().emplace_back(_build_partition_update());
- }
if (_file_format_transformer != nullptr) {
Status st = _file_format_transformer->close();
if (!st.ok()) {
@@ -156,6 +153,9 @@ Status VHivePartitionWriter::close(const Status& status) {
LOG(WARNING) << fmt::format("Delete file {} failed, reason: {}",
path, st.to_string());
}
}
+ if (status.ok()) {
+
_state->hive_partition_updates().emplace_back(_build_partition_update());
+ }
return Status::OK();
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
