voxuannguyen2001 opened a new issue, #65848: URL: https://github.com/apache/doris/issues/65848
### Search before asking - [x] I searched the [issues](https://github.com/apache/doris/issues) and found no similar issue. ### Version Apache Doris **4.1.1** (docker image `dyrnq/doris:4.1.1`). Also reproduced identically on **4.1.0** and **4.1.2** — not a regression within 4.1.x. ### What's Wrong? When a client uses the MySQL **binary prepared-statement protocol** with statement caching (i.e. it omits the parameter TYPE bytes on repeat executes — `new-params-bound-flag = 0`) and sends a prepared **write** (`INSERT`) to a **non-master (follower) FE**, the execute is forwarded to the master FE via `FrontendServiceImpl.forward`. On the master, the forwarded execute has **no placeholder types**, so `Placeholder.getMysqlColType()` calls `Optional.get()` on an empty `Optional` and throws: ``` java.util.NoSuchElementException: No value present at java.base/java.util.Optional.get(Optional.java:143) at org.apache.doris.nereids.trees.expressions.Placeholder.getMysqlColType(Placeholder.java:97) at org.apache.doris.qe.MysqlConnectProcessor.handleExecute(MysqlConnectProcessor.java:156) at org.apache.doris.qe.ConnectProcessor.proxyExecute(ConnectProcessor.java:753) at org.apache.doris.service.FrontendServiceImpl.forward(FrontendServiceImpl.java:1137) at ... ``` The client sees: `{errorMessage=NoSuchElementException, msg: No value present, errorCode=1105, sqlState=HY000}`. Key characteristics: - **Per-execute, not per-connection.** The **first** execute on a connection carries the placeholder type bytes and forwards fine; every **cached** repeat execute omits them and fails. In a 30-execute loop on one connection we consistently get `OK=1, failures=29`. - **Master vs follower.** The exact same client and loop against the **master** FE succeeds 30/30. Only executes **forwarded from a follower** fail. This shows Doris handles `new-params-bound-flag = 0` correctly when the FE that ran `COM_STMT_PREPARE` also runs the execute — it only breaks across the forward. - **The write never happens** — it fails on the master before planning/execution. This is deployment-mode-independent (reproduced in coupled/local mode); the defect is in the FE prepared-statement forward path, not in storage. ### What You Expected? A prepared `INSERT` sent to a follower FE should behave exactly like one sent to the master — the forwarded execute should carry (or the master should recover) the placeholder types, and the write should succeed. Omitting parameter types on repeat executes is a legal MySQL binary-protocol optimization, so the master's forward path must tolerate it. At the very least, the server should never answer a legal protocol packet with an internal `NoSuchElementException` from an unguarded `Optional.get()` — it should return a clean, retryable error. The current behavior makes prepared writes fail ~non-deterministically on any multi-FE cluster (rate depends on how the connection pool mixes fresh vs cached executes and how it routes across FEs). ### How to Reproduce? **1. A cluster with a follower FE** (1 master FE + 1 follower FE + 1 BE). Any topology where the client can connect to a non-master FE works. **2. A minimal table:** ```sql CREATE DATABASE IF NOT EXISTS repro; CREATE TABLE IF NOT EXISTS repro.t ( k INT NOT NULL, v VARCHAR(50) ) DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1"); ``` **3. A client that omits parameter types on cached executes.** The vert.x reactive MySQL client with `setCachePreparedStatements(true)` does this (it caches the prepared statement per connection and sends `new-params-bound-flag = 0` on repeat executes for the same SQL): ```java // io.vertx:vertx-mysql-client:4.5.11 MySQLConnectOptions opts = new MySQLConnectOptions() .setHost(host).setPort(FOLLOWER_QUERY_PORT).setUser("root").setDatabase("repro") .setCachePreparedStatements(true).setSslMode(SslMode.DISABLED); SqlConnection conn = MySQLConnection.connect(vertx, opts).toCompletionStage().toCompletableFuture().get(); for (int i = 0; i < 30; i++) { // First iteration sends placeholder types -> forwarded OK. // Every later iteration reuses the cached statement and omits types -> forwarded execute fails on master. conn.preparedQuery("INSERT INTO repro.t (k, v) VALUES (?, ?)") .execute(Tuple.of(i, "v" + i)) .toCompletionStage().toCompletableFuture().get(); } ``` **Result:** | Target FE | Outcome | |---|---| | **Master** | 30/30 succeed | | **Follower** | **1/30 succeed, 29/30 fail** with `NoSuchElementException` / errCode 1105 (the stack above appears in the master FE's `fe.log`) | **Mitigation that confirms the mechanism:** using a **fresh connection per insert** (so every execute is a "first execute" that carries types) → **0 failures** against the follower. ### Anything Else? **Client dependency (important for triage).** The bug only manifests with clients that omit the parameter TYPE bytes on cached executes. We verified against the same follower: | Client | Behavior on repeat execute | Reproduces? | |---|---|---| | vert.x mysql-client 4.5.11 (`cachePreparedStatements=true`) | omits types (`new-params-bound-flag=0`) | **Yes** | | MySQL Connector/J 8.4 (`useServerPrepStmts=true&cachePrepStmts=true`), single prepared execute | re-sends types every execute | No | | MySQL Connector/J 8.4, server-prepared **batch** (`rewriteBatchedStatements=false`) | re-sends types every execute | No | | mysql-connector-python (`prepared=True`) | re-sends types every execute | No | So the trigger is the standard, spec-legal MySQL optimization of not re-sending types when they haven't changed. The clients that re-send types every execute happen to mask it. The master's forward path should tolerate the type-omitted case. **Likely origin.** The non-master prepared-statement forwarding path appears to come from the feature that added prepared-statement forwarding to the master (#48689, built on Nereids prepared statements in #35318). #63920 (present in 4.1.2) touches the prepared-statement path but is point-query(read)-only and does **not** cover this forwarded-INSERT case (verified: 4.1.2 still reproduces). Root cause looks like `MysqlConnectProcessor.handleExecute` / `ConnectProcessor.proxyExecute` not carrying the placeholder MySQL column types across the `forward` RPC, so on the master `Placeholder.getMysqlColType()` finds an empty `Optional`. **Suggested fix directions:** either (a) carry the placeholder types in the forwarded request so the master can rebuild them, or (b) have the master re-derive types from the bound values / fall back gracefully instead of `Optional.get()`, or at minimum (c) return a clear, retryable error instead of an internal `NoSuchElementException`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
