JackieTien97 commented on code in PR #19269:
URL: https://github.com/apache/iotdb/pull/19269#discussion_r4564785792
##########
iotdb-client/service-rpc/src/main/java/org/apache/iotdb/rpc/TElasticFramedTransport.java:
##########
@@ -288,10 +287,24 @@ public void checkReadBytesAvailable(long numBytes) throws
TTransportException {
}
@Override
- public void write(byte[] buf, int off, int len) {
+ public void write(byte[] buf, int off, int len) throws TTransportException {
+ long size = (long) writeBuffer.getPos() + len;
+ int maxSize = getMaxWriteBufferSize();
+ if (size > maxSize) {
+ // Thrift serializes the whole request before flush. Reject it before
growing the buffer so
+ // an oversized request cannot keep allocating and retain the caller's
query memory.
+ close();
+ throw new TTransportException(
+ TTransportException.CORRUPTED_DATA,
+ "Frame size (" + size + ") larger than protect max size (" + maxSize
+ ")!");
Review Comment:
**[P3] The reported size is now just past the limit, not the request size**
For the same 64-device query:
- base logged `Frame size (3811712) larger than protect max size (1048576)!`;
- this head logs `Frame size (1048582) larger than protect max size
(1048576)!`, which is also what the client now sees in the 305 message.
The old number showed how far over the limit the request was, which is what
an operator needs in order to size `dn_thrift_max_frame_size`. The full size
isn't known at this point anymore. Consider wording the message as "at least N
bytes" and naming the setting.
##########
iotdb-client/service-rpc/src/main/java/org/apache/iotdb/rpc/TElasticFramedTransport.java:
##########
@@ -288,10 +287,24 @@ public void checkReadBytesAvailable(long numBytes) throws
TTransportException {
}
@Override
- public void write(byte[] buf, int off, int len) {
+ public void write(byte[] buf, int off, int len) throws TTransportException {
+ long size = (long) writeBuffer.getPos() + len;
+ int maxSize = getMaxWriteBufferSize();
+ if (size > maxSize) {
+ // Thrift serializes the whole request before flush. Reject it before
growing the buffer so
+ // an oversized request cannot keep allocating and retain the caller's
query memory.
+ close();
+ throw new TTransportException(
+ TTransportException.CORRUPTED_DATA,
+ "Frame size (" + size + ") larger than protect max size (" + maxSize
+ ")!");
+ }
writeBuffer.write(buf, off, len);
Review Comment:
**[P3] Growth can still overshoot the limit before the rejection**
`writeBuffer.write` grows the buffer by 1.5× without knowing
`getMaxWriteBufferSize()`. The last growth step before the rejection can
therefore allocate well past the limit. Measured on this head with 8-byte
writes and a 1024-byte initial buffer:
| limit | capacity at rejection |
|---|---|
| 1 MiB | 1,513,342 (1.44×) |
| 64 MiB (DataNode default for heaps ≥ 4 GiB) | 87,266,902 (1.30×) |
Passing the cap down to the write buffer would keep the array at or below
the limit, e.g. `min(grow, maxSize)` when growing.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/Coordinator.java:
##########
@@ -162,6 +165,11 @@ private ExecutionResult execution(
result.status.setNeedRetry(true);
}
return result;
+ } catch (RuntimeException | Error e) {
+ // Analysis can fail before the execution is registered, so later error
handling cannot look
+ // up its SQL in queryExecutionMap. Keep the SQL and query ID in the
failure log here.
+ LOGGER.warn("[QueryFailed] queryId: {}, sql: {}", globalQueryId, sql, e);
Review Comment:
**[P2] This logs every analysis/planning failure at WARN with a stack trace,
including user errors that `ErrorHandlingUtils` deliberately keeps at INFO**
`ErrorHandlingUtils#onQueryException` already logs these failures:
- user-caused ones (`SQL_PARSE_ERROR`, `SEMANTIC_ERROR`, `NO_PERMISSION`,
`QUERY_TIMEOUT`, …) at INFO with no stack trace;
- everything else with the stack.
On the client path that line already includes the SQL: `Query Statement:
"<sql>". executeStatement failed because …`. This catch runs before that
classification. DataNode log on this head, client statements via JDBC:
| statement | existing `ErrorHandlingUtils` line | new `[QueryFailed]` line |
|---|---|---|
| `select count(s1), s1 from root.db.d0` → 701 | INFO, no stack | WARN +
`SemanticException` stack (23 frames) |
| oversized partition request → 305 | WARN + stack | WARN + the same stack
again |
| `select s1 from root.db.d0 where time >= 0 and time <=
9223372036854775806` → 1707 | WARN + stack | WARN + the same stack again |
The consequences:
- A dashboard that keeps sending an invalid query now writes a WARN stack
trace on every execution.
- So does a query that hits the new analysis-time deadline check
(`QueryTimeoutRuntimeException`).
- On the client path, every other failure that throws out of `execution()`
is logged twice.
If the goal is to tie the query ID to the SQL when a failure happens before
registration, two options:
- Log this line without the throwable, since callers already log it.
- Skip the classes that `ErrorHandlingUtils` treats as user errors (at least
`SemanticException` and `QueryTimeoutRuntimeException`), or log those at DEBUG.
`assertFailureLogs` in `QueryPartitionMemoryTest` would need to follow
whichever option is chosen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]