This is an automated email from the ASF dual-hosted git repository.

Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
     new 4befc5bfe79 [fix](arrow-flight) Preserve query errors after result 
buffer cancellation (#70676)
4befc5bfe79 is described below

commit 4befc5bfe79e7bffd0ac71818e324068637cc011
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Wed Nov 18 10:12:20 2026 +0800

    [fix](arrow-flight) Preserve query errors after result buffer cancellation 
(#70676)
    
    ### Background
    
    Flight SQL fetches a result schema from the BE before the client reads
    its rows. The BE registers result buffers by query or fragment instance
    ID and releases unread data when a query is aborted.
    
    ### What problem does this PR solve?
    
    Issue Number: None
    Related PR: #69945 (introduced the regression), #69996 (query
    cancellation across backends), #67644 (separate handling of missing
    buffers)
    PR Type: Bugfix
    
    Problem Summary:
    
    A failed Flight query could report "no arrow schema for this query" when
    cancellation removed its result buffer before schema retrieval. Preserve
    the original failure status until timed cleanup while still releasing
    unread rows immediately. Most of the patch tests failure ordering,
    reclamation and actual Flight error propagation.
    
    For example, this query now consistently reports the original percentile
    argument error in the Flight regression:
    
    ```sql
    SELECT percentile(number, -1) FROM numbers("number"="1000");
    ```
    
    The failure when cancellation preceded schema retrieval was reproduced
    before the fix with a deterministic native schema RPC test.
    
    ### Details
    
    - **Frequency:** Intermittent when cancellation completes before schema
    retrieval; the native test forces this ordering.
    - **Root cause:** `cancel_arrow_flight_buffers` removed the buffer
    registration and access to its failure status together. The later schema
    RPC failed in `find_buffer` before reading that status.
    - **Why existing tests missed it:** The Flight cancellation unit tests
    checked early-close cleanup, waiting readers and successful termination,
    but never fetched schema after failure cancellation.
    `test_aggregate_all_functions` checks percentile error text without
    controlling the ordering.
    - **Fix:** Save only the terminal status under the lock for the buffer
    map before removing each buffer. Return it for late lookups and reclaim
    it through the existing cancellation thread, including when sink failure
    skips normal cleanup during close. Protect schema status reads with the
    buffer mutex.
    - **Other paths and branches:** Cover query and instance IDs, sibling
    buffers, repeated cancellation and expiry without sink close. Existing
    LIMIT_REACH, FINISHED, normal EOF and early-close tests still pass.
    branch-4.1 and branch-4.2 have the same pattern from #68630; branch-3.1
    and branch-4.0 lack this cancellation path. No RPC fields, defaults or
    session variables change.
    
    ### Testing
    
    - Native baseline: 21 tests passed; only the new
    cancellation-before-schema test failed, returning code 6 instead of 33
    and losing the original message. With the fix, all 31 targeted tests
    passed in each of three repetitions (93 executions).
    - A compiled mutant omitting status-expiry scheduling failed the new
    reclamation test; restoring the candidate passed again.
    - Real local FE + BE: `test_flight_query_error_after_cancel` passed with
    80 expected execution errors and 40 successful following queries across
    both result sink modes. `test_flight_cancel_cleanup` and
    `test_aggregate_all_functions` also passed through Flight.
    
    ### Notes for Reviewers
    
    Native and runtime tests used a compatible local build baseline with
    identical changed files. The task sources also passed compilation checks
    against regenerated protocol headers from current master. A complete
    build of current master was not run because the installed dependency
    package lacks MongoDB C Driver 2.0. Generic handling of missing buffers
    in #67644 is separate from preserving the original query status here.
    
    ### Release note
    
    Flight SQL returns the original execution error when a failed query is
    cancelled before its result schema is fetched.
    
    ### Check List (For Author)
    
    - Behavior changed: No
    - Does this need documentation: No
    - Pre-submission status: Partially verified: native and runtime tests
    used a matching compatibility baseline; a complete build of current
    master was not run locally.
    
    ### AI Disclosure
    
    - Tool: Codex (GPT-6)
    - Generated: Implementation, tests, and description.
    - Human review: The author has not reviewed the generated code.
---
 .../sink/writer/varrow_flight_result_writer.cpp    |  1 +
 be/src/runtime/result_buffer_mgr.cpp               | 26 +++++--
 be/src/runtime/result_buffer_mgr.h                 |  2 +
 .../arrow_flight/arrow_flight_cancel_test.cpp      | 87 ++++++++++++++++++++++
 .../test_flight_query_error_after_cancel.groovy    | 45 +++++++++++
 5 files changed, 156 insertions(+), 5 deletions(-)

diff --git a/be/src/exec/sink/writer/varrow_flight_result_writer.cpp 
b/be/src/exec/sink/writer/varrow_flight_result_writer.cpp
index 867a8608c2b..dbbb5b7eec6 100644
--- a/be/src/exec/sink/writer/varrow_flight_result_writer.cpp
+++ b/be/src/exec/sink/writer/varrow_flight_result_writer.cpp
@@ -82,6 +82,7 @@ Status GetArrowResultBatchCtx::on_data(const 
std::shared_ptr<Block>& block,
 }
 
 Status 
ArrowFlightResultBlockBuffer::get_schema(std::shared_ptr<arrow::Schema>* 
arrow_schema) {
+    std::lock_guard<std::mutex> lock(_lock);
     if (!_status.ok()) {
         return _status;
     }
diff --git a/be/src/runtime/result_buffer_mgr.cpp 
b/be/src/runtime/result_buffer_mgr.cpp
index 177b806c15a..b8728dd70bc 100644
--- a/be/src/runtime/result_buffer_mgr.cpp
+++ b/be/src/runtime/result_buffer_mgr.cpp
@@ -30,6 +30,7 @@
 #include <utility>
 
 #include "arrow/type_fwd.h"
+#include "common/config.h"
 #include "common/metrics/doris_metrics.h"
 #include "common/metrics/metrics.h"
 #include "common/status.h"
@@ -125,15 +126,25 @@ template <typename ResultBlockBufferType>
 Status ResultBufferMgr::find_buffer(const TUniqueId& finst_id,
                                     std::shared_ptr<ResultBlockBufferType>& 
buffer) {
     buffer = _find_control_block<ResultBlockBufferType>(finst_id);
-    return buffer == nullptr ? Status::InternalError(
-                                       "no arrow schema for this query, maybe 
query has been "
-                                       "canceled, finst_id={}",
-                                       print_id(finst_id))
-                             : Status::OK();
+    if (buffer == nullptr) {
+        // A failed fragment can be cancelled before the FE fetches its 
schema. The data buffer
+        // is already released, but this request must still receive the 
query's original error.
+        std::shared_lock<std::shared_mutex> rlock(_buffer_map_lock);
+        auto cancelled = _cancelled_arrow_flight_statuses.find(finst_id);
+        if (cancelled != _cancelled_arrow_flight_statuses.end()) {
+            return cancelled->second;
+        }
+        return Status::InternalError(
+                "no arrow schema for this query, maybe query has been "
+                "canceled, finst_id={}",
+                print_id(finst_id));
+    }
+    return Status::OK();
 }
 
 bool ResultBufferMgr::cancel(const TUniqueId& unique_id, const Status& reason) 
{
     std::unique_lock<std::shared_mutex> wlock(_buffer_map_lock);
+    _cancelled_arrow_flight_statuses.erase(unique_id);
     auto iter = _buffer_map.find(unique_id);
 
     auto exist = _buffer_map.end() != iter;
@@ -173,6 +184,11 @@ void ResultBufferMgr::cancel_arrow_flight_buffers(const 
TUniqueId& query_id, con
             auto it = _buffer_map.find(buffer_id);
             DORIS_CHECK(it != _buffer_map.end());
             
buffers.push_back(std::static_pointer_cast<ArrowFlightResultBlockBuffer>(it->second));
+            _cancelled_arrow_flight_statuses.emplace(buffer_id, reason);
+            // A failed sink's close can return before scheduling its usual 
deferred cleanup.
+            // Bound the lifetime of the saved error even when no client 
fetches the schema.
+            cancel_at_time(time(nullptr) + 
config::result_buffer_cancelled_interval_time,
+                           buffer_id);
             _buffer_map.erase(it);
         }
         _arrow_flight_query_buffers.erase(query);
diff --git a/be/src/runtime/result_buffer_mgr.h 
b/be/src/runtime/result_buffer_mgr.h
index e0170c88176..53d5efa67b1 100644
--- a/be/src/runtime/result_buffer_mgr.h
+++ b/be/src/runtime/result_buffer_mgr.h
@@ -94,6 +94,8 @@ private:
     // buffer block map
     BufferMap _buffer_map;
     std::unordered_map<TUniqueId, std::unordered_set<TUniqueId>> 
_arrow_flight_query_buffers;
+    // Keep only the failure after releasing unread blocks, until the normal 
timed cleanup.
+    std::unordered_map<TUniqueId, Status> _cancelled_arrow_flight_statuses;
 
     // lock for timeout map
     std::mutex _timeout_lock;
diff --git a/be/test/service/arrow_flight/arrow_flight_cancel_test.cpp 
b/be/test/service/arrow_flight/arrow_flight_cancel_test.cpp
index 3be5203e958..8a5a4005d39 100644
--- a/be/test/service/arrow_flight/arrow_flight_cancel_test.cpp
+++ b/be/test/service/arrow_flight/arrow_flight_cancel_test.cpp
@@ -41,6 +41,7 @@
 #include "util/brpc_client_cache.h"
 #include "util/client_cache.h"
 #include "util/debug/leakcheck_disabler.h"
+#include "util/defer_op.h"
 #include "util/dns_cache.h"
 
 namespace doris::flight {
@@ -134,6 +135,56 @@ TEST_F(ArrowFlightCancelTest, 
CloseReleasesSiblingBuffersAfterContextExpires) {
     EXPECT_TRUE(arrow_sibling->_result_batch_queue.empty());
 }
 
+TEST_F(ArrowFlightCancelTest, CancelledStatusCoversQueryAndInstanceIds) {
+    fill_buffer();
+    std::shared_ptr<ResultBlockBufferBase> parallel_buffer;
+    ASSERT_TRUE(_mgr.create_sender(_state.query_id(), 16, &parallel_buffer, 
&_state, true,
+                                   arrow::schema({arrow::field("value", 
arrow::int64())}))
+                        .ok());
+    const auto failure = Status::InvalidArgument("failed before fetching 
schema");
+    _mgr.cancel_arrow_flight_buffers(_state.query_id(), failure);
+    EXPECT_TRUE(_buffer->_result_batch_queue.empty());
+    EXPECT_TRUE(_dep->ready());
+    EXPECT_TRUE(_mgr._buffer_map.empty());
+    EXPECT_TRUE(_mgr._arrow_flight_query_buffers.empty());
+    for (const auto& id : {_id, _state.query_id()}) {
+        std::shared_ptr<ArrowFlightResultBlockBuffer> found;
+        const auto status = _mgr.find_buffer(id, found);
+        EXPECT_EQ(status.code(), failure.code());
+        EXPECT_EQ(status.msg(), failure.msg());
+        EXPECT_EQ(found, nullptr);
+    }
+}
+
+TEST_F(ArrowFlightCancelTest, 
RepeatedCancellationPreservesFirstErrorUntilCleanup) {
+    const auto failure = Status::InvalidArgument("original execution failure");
+    _mgr.cancel_arrow_flight_buffers(_state.query_id(), failure);
+    _mgr.cancel_arrow_flight_buffers(_state.query_id(), 
Status::Cancelled("later cancellation"));
+    std::shared_ptr<ArrowFlightResultBlockBuffer> found;
+    const auto status = _mgr.find_buffer(_id, found);
+    EXPECT_EQ(status.code(), failure.code());
+    EXPECT_EQ(status.msg(), failure.msg());
+    EXPECT_FALSE(_mgr.cancel(_id, Status::Cancelled("expired result")));
+    EXPECT_NE(_mgr.find_buffer(_id, found).msg().find("no arrow schema"), 
std::string::npos);
+}
+
+TEST_F(ArrowFlightCancelTest, CancelledStatusExpiresWithoutSinkClose) {
+    const int32_t previous_interval = 
config::result_buffer_cancelled_interval_time;
+    config::result_buffer_cancelled_interval_time = 0;
+    Defer restore {[&] { config::result_buffer_cancelled_interval_time = 
previous_interval; }};
+    const auto failure = Status::InvalidArgument("unclaimed query failure");
+    _mgr.cancel_arrow_flight_buffers(_state.query_id(), failure);
+    ASSERT_TRUE(_mgr.init().ok());
+    std::shared_ptr<ArrowFlightResultBlockBuffer> found;
+    for (int attempt = 0; attempt < 200; ++attempt) {
+        if (_mgr.find_buffer(_id, found).msg().find(failure.msg()) == 
std::string::npos) {
+            break;
+        }
+        std::this_thread::sleep_for(std::chrono::milliseconds(10));
+    }
+    EXPECT_NE(_mgr.find_buffer(_id, found).msg().find("no arrow schema"), 
std::string::npos);
+}
+
 TEST_F(ArrowFlightCancelTest, NormalEofIsNotCancellation) {
     bool fully_closed = false;
     ASSERT_TRUE(_buffer->close(_state.fragment_instance_id(), Status::OK(), 0, 
fully_closed).ok());
@@ -651,6 +702,42 @@ TEST_F(ArrowFlightRemoteCancelTest, 
SuccessfulFragmentStopPreservesUnreadResults
     }
 }
 
+TEST_F(ArrowFlightRemoteCancelTest, 
FragmentFailurePreservesErrorForLateSchemaFetch) {
+    fill_buffer();
+    const auto failure = Status::InvalidArgument("percentile parameter must 
not be -1");
+    auto stub = _cache.get_client(_statement->result_addr);
+    PCancelPlanFragmentRequest cancel_request;
+    cancel_request.mutable_finst_id()->set_hi(_id.hi);
+    cancel_request.mutable_finst_id()->set_lo(_id.lo);
+    cancel_request.mutable_query_id()->set_hi(_state.query_id().hi);
+    cancel_request.mutable_query_id()->set_lo(_state.query_id().lo);
+    failure.to_protobuf(cancel_request.mutable_cancel_status());
+    PCancelPlanFragmentResult cancel_response;
+    brpc::Controller cancel_controller;
+    cancel_controller.set_timeout_ms(1000);
+    stub->cancel_plan_fragment(&cancel_controller, &cancel_request, 
&cancel_response, nullptr);
+    ASSERT_FALSE(cancel_controller.Failed()) << cancel_controller.ErrorText();
+    ASSERT_TRUE(Status::create(cancel_response.status()).ok());
+    EXPECT_TRUE(_buffer->_result_batch_queue.empty());
+    EXPECT_TRUE(_dep->ready());
+
+    // Wait for cancellation to complete before fetching schema, so the 
failure ordering is
+    // deterministic rather than depending on the relative speed of the two BE 
work pools.
+    PFetchArrowFlightSchemaRequest schema_request;
+    schema_request.mutable_finst_id()->set_hi(_id.hi);
+    schema_request.mutable_finst_id()->set_lo(_id.lo);
+    PFetchArrowFlightSchemaResult schema_response;
+    brpc::Controller schema_controller;
+    schema_controller.set_timeout_ms(1000);
+    stub->fetch_arrow_flight_schema(&schema_controller, &schema_request, 
&schema_response, nullptr);
+    ASSERT_FALSE(schema_controller.Failed()) << schema_controller.ErrorText();
+    ASSERT_TRUE(schema_response.has_status());
+    const auto schema_status = Status::create(schema_response.status());
+    EXPECT_EQ(schema_status.code(), failure.code()) << 
schema_status.to_string();
+    // Status serialization adds the backend address and status code to the 
message.
+    EXPECT_NE(schema_status.msg().find(failure.msg()), std::string::npos);
+}
+
 TEST_F(ArrowFlightRemoteCancelTest, NormalEofDoesNotCancelQuery) {
     bool fully_closed = false;
     ASSERT_TRUE(_buffer->close(_state.fragment_instance_id(), Status::OK(), 0, 
fully_closed).ok());
diff --git 
a/regression-test/suites/arrow_flight_sql_p0/test_flight_query_error_after_cancel.groovy
 
b/regression-test/suites/arrow_flight_sql_p0/test_flight_query_error_after_cancel.groovy
new file mode 100644
index 00000000000..bb905b32cf6
--- /dev/null
+++ 
b/regression-test/suites/arrow_flight_sql_p0/test_flight_query_error_after_cancel.groovy
@@ -0,0 +1,45 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_flight_query_error_after_cancel", "arrow_flight_sql") {
+    def originalConnection = context.threadLocalConn.get()
+    context.getArrowFlightSqlConnection()
+    // TestAction uses getConnection() even when the suite runs in the Flight 
group.
+    // Reuse the owned Flight connection so exception assertions always 
exercise Flight.
+    context.threadLocalConn.set(context.threadArrowFlightSqlConn.get())
+    try {
+        for (def parallelResultSink : [false, true]) {
+            sql "SET enable_parallel_result_sink=${parallelResultSink}"
+            for (int iteration = 0; iteration < 20; ++iteration) {
+                for (def quantile : [-1, 3000]) {
+                    test {
+                        sql "SELECT percentile(number, ${quantile}) FROM 
numbers(\"number\"=\"1000\")"
+                        exception "quantile in func percentile should in [0, 
1], but real data is:${quantile}"
+                    }
+                }
+                // A failed query must not poison the connection's next BE 
result stream.
+                sql "SELECT count(*) FROM numbers(\"number\"=\"1000\")"
+            }
+        }
+    } finally {
+        if (originalConnection == null) {
+            context.threadLocalConn.remove()
+        } else {
+            context.threadLocalConn.set(originalConnection)
+        }
+    }
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to