Hi all,
I'd like to raise a design for discussion before submitting the implementation. Problem: A single row containing an OBJECT/BLOB field can reach hundreds of MB. The DataNode writes the whole serialized TsBlock into one TSFetchResultsResp.queryResult (or TSExecuteStatementResp.queryResult), and the existing limits (fetchSize in rows, MAX_RETURN_SIZE = 8MB) cannot cap a single oversized row. So: - The response frame is as large as the oveversized row; - TElasticFramedTransport grows its read/write buffers to the frame size and keeps them for the lifetime of the connection — one such response leaves roughly the row size resident on each of the server connection thread and the client, per active connection, which easily cause OOM. Proposal: We recently solved the same problem for DataNode-to-DataNode exchange (MPPDataExchange large TsBlock fragmentation). This proposal applies the same idea to the client-server path, simplified to a pure pull model: - Add optional offset/totalLength fields to TSFetchResultsReq/TSFetchResultsResp (and the two fields to TSExecuteStatementResp). A client that sets offset (0 on the first request) declares it can assemble fragments; old clients and servers keep the existing behavior. - The server cuts a serialized TsBlock into fragments that fit a bounded payload (reuses the existing conf `mpp_data_exchange_max_payload_size_in_bytes`); the unsent tail is kept on QueryExecution and continued by the next request. Fragments are zero-copy slice() views; a TsBlock is still serialized only once. - IoTDBRpcDataSet (shared by Session and JDBC) reassembles fragments before TsBlockSerde.deserialize, so SessionDataSet, RowRecord, Field and JDBC users are unaffected. - Small results take exactly the same RPCs as today; only an oversized block costs one additional round trip, because the response that completes the fragment does not piggyback new rows (this keeps the per-response row count within the existing fetchSize bound). Compatibility. Protocol changes are purely additive optional fields, so rolling upgrades and other language clients are unaffected. The execute path negotiates via an enable_result_chunking key in the openSession configuration map. Thanks, Weihao Li
