JackieTien97 commented on code in PR #19278:
URL: https://github.com/apache/iotdb/pull/19278#discussion_r4571779722


##########
iotdb-client/service-rpc/src/main/java/org/apache/iotdb/rpc/AutoScalingBufferWriteTransport.java:
##########
@@ -55,17 +57,48 @@ public int read(byte[] buf, int off, int len) {
 
   @Override
   public void write(byte[] toWrite, int off, int len) throws 
TTransportException {
+    write(toWrite, off, len, AutoResizingBuffer.MAX_ARRAY_SIZE);
+  }
+
+  void write(byte[] toWrite, int off, int len, int maxCapacity) throws 
TTransportException {
+    int limit = Math.min(maxCapacity, AutoResizingBuffer.MAX_ARRAY_SIZE);
+    long requiredCapacity = (long) pos + len;
+    if (requiredCapacity > limit) {
+      throw new TTransportException(
+          TTransportException.CORRUPTED_DATA,
+          String.format(
+              RpcMessages
+                  
.EXCEPTION_WRITE_BUFFER_SIZE_ARG_LARGER_THAN_MAXIMUM_BUFFER_SIZE_ARG_C0A47F3D,
+              requiredCapacity,
+              limit));
+    }
+    // A small prefix during serialization does not indicate a small frame. 
Only the complete
+    // frame size, supplied after flushing, may trigger shrinking.
     try {
-      // A small prefix during serialization does not indicate a small frame. 
Only the complete
-      // frame size, supplied after flushing, may trigger shrinking.
-      ensureCapacity(pos + len);
+      ensureCapacity((int) requiredCapacity, limit);
     } catch (IOException e) {
       throw new TTransportException(e);
     }
     System.arraycopy(toWrite, off, getBuffer(), pos, len);
     pos += len;
   }
 
+  // Framed callers keep capacity within their limit. A fitting write needs no 
further limit
+  // checks; compare remaining space rather than adding pos and len, which 
could overflow.
+  boolean tryWriteWithinCapacity(byte[] toWrite, int off, int len) {
+    byte[] array = getBuffer();
+    // A closed reusable buffer must reserve its memory again. Crossing the 
reuse threshold
+    // must also go through ensureCapacity so the large frame uses temporary 
storage.
+    if (buf.isClosed()
+        || len > array.length - pos
+        || (temporaryBuffer == null && len > maxReusableBufferSize - pos)) {

Review Comment:
   **Nit: no test pins the reuse-threshold exit**
   
   This clause is needed. The reusable array can already be larger than 
`maxReusableBufferSize`, because 1.5× growth from below the threshold 
overshoots it. Without the clause, a write that crosses the threshold but still 
fits would stay in the reusable, memory-accounted buffer instead of moving to 
the temporary one. The buffer would then keep that capacity after `flush()`.
   
   No test catches its removal, though. I deleted only this clause, and all 95 
tests still passed in `AutoResizingBufferTest`, `TElasticFramedTransportTest`, 
`TCompressedElasticFramedTransportTest`, `TElasticFramedTransportReadTest` and 
`DeepCopyRpcTransportFactoryTest`. By contrast, removing `buf.isClosed()` does 
fail `testFramedFastPathReservesMemoryAfterReopening`.
   
   Probe setup: initial capacity 8, frame limit 100, threshold 32. Four 8-byte 
writes grow the reusable array 8 → 16 → 24 → 36. One 4-byte write then takes 
the frame to 36:
   
   ```
   this head:       after the crossing write: reusable=8, temporary=36   after 
flush: reusable=8
   clause removed:  after the crossing write: reusable=36, no temporary  after 
flush: reusable=36
   ```
   
   `testTemporaryWriteBufferGrowthRespectsFrameLimit` writes in 8-byte steps, 
so it never reaches this crossing-but-fitting case. A test with the sequence 
above would lock the clause in. It could assert the reusable capacity, or 
`memoryControl.getUsedMemoryInBytes()`, after the crossing write and after 
`flush()`.



##########
.github/workflows/sonar-codecov.yml:
##########
@@ -50,6 +50,11 @@ jobs:
 
     steps:
       - uses: actions/checkout@v5
+      - name: Set up JDK 21
+        uses: actions/setup-java@v5
+        with:
+          distribution: corretto
+          java-version: 21

Review Comment:
   **Nit: this pin isn't needed on master**
   
   On dev/1.3 the pin fixed a `codecov` failure on every push: spotless threw 
`NoSuchMethodError` under the runner's default JDK. On master the `codecov` job 
already passes without it. In the push run for `e78f736613` (run 43417743453), 
`codecov` succeeded; only the `sonar` job failed, at the SonarCloud Report step.
   
   The pin is harmless. Dropping it keeps this port to the code change, while 
keeping it keeps the two branches' workflows aligned. Your call.



##########
iotdb-client/service-rpc/src/main/java/org/apache/iotdb/rpc/TElasticFramedTransport.java:
##########
@@ -383,7 +386,29 @@ public void resetMessageSizeAndConsumedBytes(long newSize) 
throws TTransportExce
 
   @Override
   public void write(byte[] buf, int off, int len) throws TTransportException {
-    writeBuffer.write(buf, off, len);
+    if (writeBuffer.tryWriteWithinCapacity(buf, off, len)) {
+      return;
+    }
+    long size = (long) writeBuffer.getPos() + len;
+    int maxSize = getMaxWriteBufferSize();
+    if (size > maxSize) {
+      // Thrift serializes the whole request before flush. Reject it before 
growing the buffer so
+      // an oversized request cannot keep allocating and retain the caller's 
query memory.
+      close();
+      String message =
+          maxSize == AutoResizingBuffer.MAX_ARRAY_SIZE
+              ? RpcMessages
+                  
.EXCEPTION_SERIALIZED_FRAME_SIZE_IS_AT_LEAST_ARG_BYTES_EXCEEDING_THE_MAXIMUM_JAVA_ARRAY_SIZE_OF_ARG_BYTES_E21E05CA
+              : RpcMessages
+                  
.EXCEPTION_SERIALIZED_FRAME_SIZE_IS_AT_LEAST_ARG_BYTES_EXCEEDING_THE_CONFIGURED_THRIFT_MAX_FRAME_SIZE_OF_ARG_BYTES_1CDD4BB9;
+      throw new TTransportException(
+          TTransportException.CORRUPTED_DATA, String.format(message, size, 
maxSize));

Review Comment:
   **[P3] The new rejection drops the remote endpoint that master's write-side 
frame error has included since #17536**
   
   `checkWriteFrameSize` (L311–325) appends `" to " + remoteAddress`; #17536 
added it to diagnose oversized fragment-instance dispatches. For uncompressed 
transports this branch now rejects before `flush()`, so that check never runs 
and the endpoint disappears from the error.
   
   Same Session probe against master base and this head 
(`thriftMaxFrameSize(4194304)`, 1,000,000 random INT64 rows):
   
   ```
   base:       TTransportException: Frame size (8310216) larger than protect 
max size (4194304) to /127.0.0.1:6973!
   this head:  TTransportException: Serialized frame size is at least 8124109 
bytes, exceeding the configured Thrift max frame size of 4194304 bytes.
   ```
   
   The DataNode → ConfigNode partition request loses `to /127.0.0.1:17710` in 
the same way.
   
   #17536's handling still works, because it keys on `CORRUPTED_DATA`: dispatch 
still maps to 308 without retry. Only the diagnostic text regresses.
   
   Suggestion: build the remote info before `close()`, as `checkWriteFrameSize` 
does (a small helper could serve both). Then add a trailing `%s` to the two new 
constants, in both en and zh, matching `FRAME_ERROR_FRAME_SIZE_EXCEEDED`:
   
   ```java
   String remoteInfo = "";
   if (underlying instanceof TSocket) {
     SocketAddress remoteAddress = ((TSocket) 
underlying).getSocket().getRemoteSocketAddress();
     remoteInfo = remoteAddress == null ? "" : " to " + remoteAddress;
   }
   close();
   // ...
   throw new TTransportException(
       TTransportException.CORRUPTED_DATA, String.format(message, size, 
maxSize, remoteInfo));
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to