Gabriel Reid created CALCITE-3333:
-------------------------------------
Summary: Add time-based of ResultSet frame size limiting
Key: CALCITE-3333
URL: https://issues.apache.org/jira/browse/CALCITE-3333
Project: Calcite
Issue Type: New Feature
Components: avatica
Reporter: Gabriel Reid
The size of a single JDBC ResultSet frame returned in a single
{{prepareAndExecute}} or {{fetch}} invocation is currently 100, meaning that
each retrieval of a portion of a ResultSet will send 100 rows over the wire.
This frame size may be too big in some situations, and too small in other
situations.
If the underlying data source being queried can provide thousands of (small)
records per second, then only reading them at 100 per RPC call will be
unnecessarily slow.
On the other hand, if the underlying data source is only providing records at a
rate of 1 per second, then it will take 100 seconds for each RPC call to
return, which can lead to timeouts (particularly if Avatica server is sitting
behind a proxy that has a strict request timeout).
The main factors to take into account when finding an ideal size of frame to
return for each RPC call are:
* make the frames small enough that they don't overload either Avatica server
or the client with overly large amounts of data at one time
* make the frames large enough so that the percentage of total query time that
is spent only on RPC overhead is minimized
The general idea of this ticket is to add a pluggable "frame size limiting"
functionality so that frame size limiting can be done based on the number of
rows, number of bytes, amount of time spent building a frame, or any other
property or combination of properties.
Note that CALCITE-2322 contains some work to allow configuring the size of a
single frame on a Connection or Statement (via the {{setFetchSize}} method),
although it's not yet merged in. That ticket would also be useful, and does not
conflict with the general intent of this ticket.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
