JackieTien97 commented on code in PR #19269:
URL: https://github.com/apache/iotdb/pull/19269#discussion_r4571068343
##########
iotdb-client/service-rpc/src/main/java/org/apache/iotdb/rpc/TElasticFramedTransport.java:
##########
@@ -288,10 +287,24 @@ public void checkReadBytesAvailable(long numBytes) throws
TTransportException {
}
@Override
- public void write(byte[] buf, int off, int len) {
+ public void write(byte[] buf, int off, int len) throws TTransportException {
+ long size = (long) writeBuffer.getPos() + len;
+ int maxSize = getMaxWriteBufferSize();
+ if (size > maxSize) {
+ // Thrift serializes the whole request before flush. Reject it before
growing the buffer so
+ // an oversized request cannot keep allocating and retain the caller's
query memory.
+ close();
+ throw new TTransportException(
+ TTransportException.CORRUPTED_DATA,
+ "Frame size (" + size + ") larger than protect max size (" + maxSize
+ ")!");
Review Comment:
Verified fixed at `c7003fff63`. The message is now generic and names the
limit that actually fired:
- **DataNode → ConfigNode, oversized partition request (305, 15–42 ms):**
`Serialized frame size is at least 1048582 bytes, exceeding the configured
Thrift max frame size of 1048576 bytes.`
- **Plain Java Session client** with `thriftMaxFrameSize(4194304)` against a
DataNode set to 1048576: `Serialized frame size is at least 8000077 bytes,
exceeding the configured Thrift max frame size of 4194304 bytes.` The number
now matches the client's own limit.
The tests now also assert that `dn_thrift_max_frame_size` is absent from the
message.
##########
iotdb-client/service-rpc/src/main/java/org/apache/iotdb/rpc/TElasticFramedTransport.java:
##########
@@ -288,10 +287,24 @@ public void checkReadBytesAvailable(long numBytes) throws
TTransportException {
}
@Override
- public void write(byte[] buf, int off, int len) {
+ public void write(byte[] buf, int off, int len) throws TTransportException {
+ long size = (long) writeBuffer.getPos() + len;
+ int maxSize = getMaxWriteBufferSize();
+ if (size > maxSize) {
+ // Thrift serializes the whole request before flush. Reject it before
growing the buffer so
+ // an oversized request cannot keep allocating and retain the caller's
query memory.
+ close();
+ throw new TTransportException(
+ TTransportException.CORRUPTED_DATA,
+ "Frame size (" + size + ") larger than protect max size (" + maxSize
+ ")!");
+ }
writeBuffer.write(buf, off, len);
Review Comment:
Verified at `c7003fff63`. The fast path doesn't change any limit behavior:
- Writes are rejected after exactly `limit` bytes (1 MiB and 64 MiB), and
capacity stays at 1.00× the limit.
- Both compressed paths still raise `TTransportException` and close the
transport.
Safety reasoning:
- `write(..., maxSize)` is the only writer of the framed `writeBuffer`, so
capacity ≤ limit holds and a fitting write can't cross the limit.
- `len > array.length - pos` can't overflow.
- `testBufferedWriteCannotBypassArrayLimitOnOverflow` would catch a `pos +
len` rewrite.
Same benchmark as before (null sink, median ns over 3 alternating JVM runs):
| | base | `827763a31d` | `c7003fff63` |
|---|---|---|---|
| small fields, ns/slot | 8.80–9.02 | 9.89–10.29 | 8.37–8.50 |
| large binary, ns/row | 1.72–1.76 | 1.70–1.73 | 1.70–1.75 |
The small-field regression is gone; this head is now about 5% faster than
base on that path. Large binary writes are unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]