JackieTien97 commented on code in PR #19278:
URL: https://github.com/apache/iotdb/pull/19278#discussion_r4571779722
##########
iotdb-client/service-rpc/src/main/java/org/apache/iotdb/rpc/AutoScalingBufferWriteTransport.java:
##########
@@ -55,17 +57,48 @@ public int read(byte[] buf, int off, int len) {
@Override
public void write(byte[] toWrite, int off, int len) throws
TTransportException {
+ write(toWrite, off, len, AutoResizingBuffer.MAX_ARRAY_SIZE);
+ }
+
+ void write(byte[] toWrite, int off, int len, int maxCapacity) throws
TTransportException {
+ int limit = Math.min(maxCapacity, AutoResizingBuffer.MAX_ARRAY_SIZE);
+ long requiredCapacity = (long) pos + len;
+ if (requiredCapacity > limit) {
+ throw new TTransportException(
+ TTransportException.CORRUPTED_DATA,
+ String.format(
+ RpcMessages
+
.EXCEPTION_WRITE_BUFFER_SIZE_ARG_LARGER_THAN_MAXIMUM_BUFFER_SIZE_ARG_C0A47F3D,
+ requiredCapacity,
+ limit));
+ }
+ // A small prefix during serialization does not indicate a small frame.
Only the complete
+ // frame size, supplied after flushing, may trigger shrinking.
try {
- // A small prefix during serialization does not indicate a small frame.
Only the complete
- // frame size, supplied after flushing, may trigger shrinking.
- ensureCapacity(pos + len);
+ ensureCapacity((int) requiredCapacity, limit);
} catch (IOException e) {
throw new TTransportException(e);
}
System.arraycopy(toWrite, off, getBuffer(), pos, len);
pos += len;
}
+ // Framed callers keep capacity within their limit. A fitting write needs no
further limit
+ // checks; compare remaining space rather than adding pos and len, which
could overflow.
+ boolean tryWriteWithinCapacity(byte[] toWrite, int off, int len) {
+ byte[] array = getBuffer();
+ // A closed reusable buffer must reserve its memory again. Crossing the
reuse threshold
+ // must also go through ensureCapacity so the large frame uses temporary
storage.
+ if (buf.isClosed()
+ || len > array.length - pos
+ || (temporaryBuffer == null && len > maxReusableBufferSize - pos)) {
Review Comment:
**Nit: no test pins the reuse-threshold exit**
This clause is needed. The reusable array can already be larger than
`maxReusableBufferSize`, because 1.5× growth from below the threshold
overshoots it. Without the clause, a write that crosses the threshold but still
fits would stay in the reusable, memory-accounted buffer instead of moving to
the temporary one. The buffer would then keep that capacity after `flush()`.
No test catches its removal, though. I deleted only this clause, and all 95
tests still passed in `AutoResizingBufferTest`, `TElasticFramedTransportTest`,
`TCompressedElasticFramedTransportTest`, `TElasticFramedTransportReadTest` and
`DeepCopyRpcTransportFactoryTest`. By contrast, removing `buf.isClosed()` does
fail `testFramedFastPathReservesMemoryAfterReopening`.
Probe setup: initial capacity 8, frame limit 100, threshold 32. Four 8-byte
writes grow the reusable array 8 → 16 → 24 → 36. One 4-byte write then takes
the frame to 36:
```
this head: after the crossing write: reusable=8, temporary=36 after
flush: reusable=8
clause removed: after the crossing write: reusable=36, no temporary after
flush: reusable=36
```
`testTemporaryWriteBufferGrowthRespectsFrameLimit` writes in 8-byte steps,
so it never reaches this crossing-but-fitting case. A test with the sequence
above would lock the clause in. It could assert the reusable capacity, or
`memoryControl.getUsedMemoryInBytes()`, after the crossing write and after
`flush()`.
##########
.github/workflows/sonar-codecov.yml:
##########
@@ -50,6 +50,11 @@ jobs:
steps:
- uses: actions/checkout@v5
+ - name: Set up JDK 21
+ uses: actions/setup-java@v5
+ with:
+ distribution: corretto
+ java-version: 21
Review Comment:
**Nit: this pin isn't needed on master**
On dev/1.3 the pin fixed a `codecov` failure on every push: spotless threw
`NoSuchMethodError` under the runner's default JDK. On master the `codecov` job
already passes without it. In the push run for `e78f736613` (run 43417743453),
`codecov` succeeded; only the `sonar` job failed, at the SonarCloud Report step.
The pin is harmless. Dropping it keeps this port to the code change, while
keeping it keeps the two branches' workflows aligned. Your call.
##########
iotdb-client/service-rpc/src/main/java/org/apache/iotdb/rpc/TElasticFramedTransport.java:
##########
@@ -383,7 +386,29 @@ public void resetMessageSizeAndConsumedBytes(long newSize)
throws TTransportExce
@Override
public void write(byte[] buf, int off, int len) throws TTransportException {
- writeBuffer.write(buf, off, len);
+ if (writeBuffer.tryWriteWithinCapacity(buf, off, len)) {
+ return;
+ }
+ long size = (long) writeBuffer.getPos() + len;
+ int maxSize = getMaxWriteBufferSize();
+ if (size > maxSize) {
+ // Thrift serializes the whole request before flush. Reject it before
growing the buffer so
+ // an oversized request cannot keep allocating and retain the caller's
query memory.
+ close();
+ String message =
+ maxSize == AutoResizingBuffer.MAX_ARRAY_SIZE
+ ? RpcMessages
+
.EXCEPTION_SERIALIZED_FRAME_SIZE_IS_AT_LEAST_ARG_BYTES_EXCEEDING_THE_MAXIMUM_JAVA_ARRAY_SIZE_OF_ARG_BYTES_E21E05CA
+ : RpcMessages
+
.EXCEPTION_SERIALIZED_FRAME_SIZE_IS_AT_LEAST_ARG_BYTES_EXCEEDING_THE_CONFIGURED_THRIFT_MAX_FRAME_SIZE_OF_ARG_BYTES_1CDD4BB9;
+ throw new TTransportException(
+ TTransportException.CORRUPTED_DATA, String.format(message, size,
maxSize));
Review Comment:
**[P3] The new rejection drops the remote endpoint that master's write-side
frame error has included since #17536**
`checkWriteFrameSize` (L311–325) appends `" to " + remoteAddress`; #17536
added it to diagnose oversized fragment-instance dispatches. For uncompressed
transports this branch now rejects before `flush()`, so that check never runs
and the endpoint disappears from the error.
Same Session probe against master base and this head
(`thriftMaxFrameSize(4194304)`, 1,000,000 random INT64 rows):
```
base: TTransportException: Frame size (8310216) larger than protect
max size (4194304) to /127.0.0.1:6973!
this head: TTransportException: Serialized frame size is at least 8124109
bytes, exceeding the configured Thrift max frame size of 4194304 bytes.
```
The DataNode → ConfigNode partition request loses `to /127.0.0.1:17710` in
the same way.
#17536's handling still works, because it keys on `CORRUPTED_DATA`: dispatch
still maps to 308 without retry. Only the diagnostic text regresses.
Suggestion: build the remote info before `close()`, as `checkWriteFrameSize`
does (a small helper could serve both). Then add a trailing `%s` to the two new
constants, in both en and zh, matching `FRAME_ERROR_FRAME_SIZE_EXCEEDED`:
```java
String remoteInfo = "";
if (underlying instanceof TSocket) {
SocketAddress remoteAddress = ((TSocket)
underlying).getSocket().getRemoteSocketAddress();
remoteInfo = remoteAddress == null ? "" : " to " + remoteAddress;
}
close();
// ...
throw new TTransportException(
TTransportException.CORRUPTED_DATA, String.format(message, size,
maxSize, remoteInfo));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]