JackieTien97 commented on code in PR #19263:
URL: https://github.com/apache/iotdb/pull/19263#discussion_r4581231285
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/exchange/source/LocalSourceHandle.java:
##########
@@ -153,7 +153,8 @@ public TsBlock receive() {
}
@Override
- public ByteBuffer getSerializedTsBlock() throws IoTDBException {
+ public ByteBuffer getSerializedTsBlock(boolean retainMemory) throws
IoTDBException {
+ // the returned bytes are a fresh serialization, not reserved from the
memory pool
Review Comment:
**[P1] Account for pending fragments on the local result path**
`receive()` removes the block from `SharedTsBlockQueue` and releases its
reservation. The serialized buffer can now outlive the RPC because
`PendingResultFragment` holds the entire buffer until the client finishes
fetching it. On a single-node deployment, several concurrent readers can
therefore hold large buffers outside the query memory pool. Please keep the
pending buffer accounted for until the last fragment is sent or the query is
cleaned up.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/ClientRPCServiceImpl.java:
##########
@@ -1570,14 +1635,15 @@ public TSFetchResultsResp
fetchResultsV2(TSFetchResultsReq req) {
statementType = queryExecution.getStatementType();
try (SetThreadName queryName = new
SetThreadName(queryExecution.getQueryId())) {
- Pair<List<ByteBuffer>, Boolean> pair =
- QueryDataSetUtils.convertQueryResultByFetchSize(queryExecution,
req.fetchSize);
- List<ByteBuffer> result = pair.left;
- finished = pair.right;
- boolean hasResultSet = !result.isEmpty();
- resp.setHasResultSet(hasResultSet);
+ QueryDataSetUtils.ResultChunk chunk =
+ fetchResultChunk(
+ queryExecution,
+ req.fetchSize,
+ req.isSetOffset() ? req.getOffset() : 0,
+ req.isSetOffset() ?
config.getMppDataExchangeMaxPayloadSizeInBytes() : 0);
+ finished = fillQueryResult(resp, chunk);
Review Comment:
**[P2] Preserve read-size quota accounting across fragments**
The execute path charges only the buffers in its response through
`quota.addReadResult()`. The continuation path here does not charge the
remaining bytes. With the default 4 MiB payload limit, a 12 MiB TsBlock that
previously fit in the execute response now counts as only 4 MiB against the
read-size quota. Please ensure the continuation bytes are accounted for exactly
once.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/ClientRPCServiceImpl.java:
##########
@@ -1455,7 +1518,9 @@ public TSExecuteStatementResp
executeFastLastDataQueryForOneDeviceV2(
regionReplicaSets.get(regionReplicaSets.size() -
1).dataNodeLocations.get(0),
clientSession.isInternalRpcSession()));
resp.setStatus(tsstatus);
- finished = SELECT_RESULT.apply(resp, queryExecution, req.fetchSize);
+ finished =
+ SELECT_RESULT.apply(
+ resp, queryExecution, req.fetchSize,
getMaxResultPayloadSize(clientSession));
Review Comment:
**[P2] Apply the payload limit to Fast LAST cache hits**
This fallback now honors the negotiated limit. The cache-hit branches in
`executeFastLastDataQueryForOneDeviceV2` and
`executeFastLastDataQueryForOnePrefixPath` still serialize the entire result
into one response and set `moreData=false`. A large cached result can still
exceed the Thrift frame limit even though the fallback can stream it in
fragments. Please apply the payload limit to both cache-hit paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]