JackieTien97 commented on code in PR #19269:
URL: https://github.com/apache/iotdb/pull/19269#discussion_r4570493339


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/Coordinator.java:
##########
@@ -162,6 +165,11 @@ private ExecutionResult execution(
         result.status.setNeedRetry(true);
       }
       return result;
+    } catch (RuntimeException | Error e) {
+      // Analysis can fail before the execution is registered, so later error 
handling cannot look
+      // up its SQL in queryExecutionMap. Keep the SQL and query ID in the 
failure log here.
+      LOGGER.warn("[QueryFailed] queryId: {}, sql: {}", globalQueryId, sql, e);

Review Comment:
   Verified fixed at `827763a31d`. I ran the same statements, and the DataNode 
log now shows:
   
   | statement | `ErrorHandlingUtils` line | `[QueryFailed]` line |
   |---|---|---|
   | `select count(s1), s1 from root.db.d0` → 701 | INFO, no stack | INFO, no 
stack |
   | oversized partition request → 305 | WARN + stack (once) | INFO, no stack |
   | `select s1 from root.db.d0 where time >= 0 and time <= 
9223372036854775806` → 1707 | WARN + stack (once) | INFO, no stack |
   
   User errors no longer produce a WARN, and each failure's stack trace is 
logged only once.



##########
iotdb-client/service-rpc/src/main/java/org/apache/iotdb/rpc/TElasticFramedTransport.java:
##########
@@ -288,10 +287,24 @@ public void checkReadBytesAvailable(long numBytes) throws 
TTransportException {
   }
 
   @Override
-  public void write(byte[] buf, int off, int len) {
+  public void write(byte[] buf, int off, int len) throws TTransportException {
+    long size = (long) writeBuffer.getPos() + len;
+    int maxSize = getMaxWriteBufferSize();
+    if (size > maxSize) {
+      // Thrift serializes the whole request before flush. Reject it before 
growing the buffer so
+      // an oversized request cannot keep allocating and retain the caller's 
query memory.
+      close();
+      throw new TTransportException(
+          TTransportException.CORRUPTED_DATA,
+          "Frame size (" + size + ") larger than protect max size (" + maxSize 
+ ")!");
+    }
     writeBuffer.write(buf, off, len);

Review Comment:
   Verified fixed at `827763a31d`. Capacity at rejection is now exactly the 
limit (1,048,576 and 67,108,864, i.e. 1.00×), and 
`testIncrementalGrowthDoesNotExceedFrameLimit` covers it.
   
   Optional perf note: these per-write checks sit on Thrift's hottest path. I 
benchmarked serialization into a null sink, taking the median of 3 alternating 
JVM runs.
   - Small fields: a 100k-slot `TDataPartitionReq`, i.e. 4 `write` calls per 
12-byte struct.
   - Large binary: a 1M-row `TSInsertTabletReq`.
   
   | | base | this head | this head + fast path below |
   |---|---|---|---|
   | small fields, ns/slot | 8.87–9.03 | 9.97–12.10 | 8.03–8.69 |
   | large binary, ns/row | 1.69–1.71 | 1.67–1.74 | 1.69–1.73 |
   
   Only small-field-heavy messages pay the cost, about 11% or more; large 
binary writes are unaffected. This commit makes `capacity <= limit` an 
invariant, so a write that fits in the current array can skip every limit 
check. The checks are only needed when the buffer must grow:
   
   ```java
   // AutoScalingBufferWriteTransport
   boolean tryWriteWithinCapacity(byte[] toWrite, int off, int len) {
     byte[] array = buf.array();
     if (len > array.length - pos) {
       return false;
     }
     System.arraycopy(toWrite, off, array, pos, len);
     pos += len;
     return true;
   }
   
   // TElasticFramedTransport#write, as the first statement
   if (writeBuffer.tryWriteWithinCapacity(buf, off, len)) {
     return;
   }
   ```
   
   With only this change, the limit behavior in my probes is unchanged: a write 
is rejected after exactly `limit` bytes, capacity equals the limit, and the 
compressed paths behave as before. The "fast path" column above is this change.



##########
iotdb-client/service-rpc/src/main/java/org/apache/iotdb/rpc/TElasticFramedTransport.java:
##########
@@ -288,10 +287,24 @@ public void checkReadBytesAvailable(long numBytes) throws 
TTransportException {
   }
 
   @Override
-  public void write(byte[] buf, int off, int len) {
+  public void write(byte[] buf, int off, int len) throws TTransportException {
+    long size = (long) writeBuffer.getPos() + len;
+    int maxSize = getMaxWriteBufferSize();
+    if (size > maxSize) {
+      // Thrift serializes the whole request before flush. Reject it before 
growing the buffer so
+      // an oversized request cannot keep allocating and retain the caller's 
query memory.
+      close();
+      throw new TTransportException(
+          TTransportException.CORRUPTED_DATA,
+          "Frame size (" + size + ") larger than protect max size (" + maxSize 
+ ")!");

Review Comment:
   The new text fixes both points: it says "at least N" and it names a setting. 
However, `TElasticFramedTransport` is the shared `service-rpc` transport, so 
the hardcoded `(dn_thrift_max_frame_size on DataNode)` is wrong outside a 
DataNode:
   - **Session/JDBC clients:** the limit is the client's own 
`Session.Builder#thriftMaxFrameSize` (default 64 MiB, 
`SessionConfig.DEFAULT_MAX_FRAME_SIZE`) or JDBC's `thrift_max_frame_size`. The 
rejection happens on the client, before anything is sent.
   - **ConfigNode:** there is no setting at all. The limit is the fixed 
`RpcUtils.THRIFT_FRAME_MAX_SIZE` (536870912), because the ConfigNode never 
loads the DataNode config. For example, a ConfigNode whose 
`iotdb-system.properties` set `dn_thrift_max_frame_size=1048576` still reported 
`protect max size (536870912)`.
   
   Repro: a plain Java client against this head's cluster (DataNode 
`dn_thrift_max_frame_size=1048576`), using `new 
Session.Builder()...thriftMaxFrameSize(4194304)` to insert a 1,000,000-row 
INT64 tablet:
   
   ```
   base client jars:      IoTDBConnectionException: 
org.apache.thrift.transport.TTransportException: Frame size (16000109) larger 
than protect max size (4194304)!
   this head's client:    IoTDBConnectionException: 
org.apache.thrift.transport.TTransportException: Serialized frame size is at 
least 8000077 bytes, exceeding the RPC frame-size limit 
(dn_thrift_max_frame_size on DataNode) of 4194304 bytes.
   ```
   
   The message names a setting that didn't fire, and that setting's actual 
value here (1048576) doesn't match the number printed. A Session bulk insert 
over the 64 MiB default would point users to the server config, which can't fix 
a client-side rejection.
   
   Suggestion: keep the library message generic, e.g. `... exceeding the 
configured Thrift max frame size of N bytes`. If the DataNode hint is wanted, 
add it where the DataNode handles the failure (e.g. the `CORRUPTED_DATA` branch 
in `ConfigNodeClient#executeRemoteCallWithRetry`), or pass a limit description 
into the transport factory. The `dn_thrift_max_frame_size` assertions in 
`TElasticFramedTransportTest` and `QueryPartitionMemoryTest` would follow. My 
earlier suggestion to name the setting didn't account for the transport being 
shared.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to