[ https://issues.apache.org/jira/browse/CALCITE-3333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated CALCITE-3333: ------------------------------------ Labels: pull-request-available (was: ) > Add time-based of ResultSet frame size limiting > ----------------------------------------------- > > Key: CALCITE-3333 > URL: https://issues.apache.org/jira/browse/CALCITE-3333 > Project: Calcite > Issue Type: New Feature > Components: avatica > Reporter: Gabriel Reid > Priority: Major > Labels: pull-request-available > > The size of a single JDBC ResultSet frame returned in a single > {{prepareAndExecute}} or {{fetch}} invocation is currently 100, meaning that > each retrieval of a portion of a ResultSet will send 100 rows over the wire. > This frame size may be too big in some situations, and too small in other > situations. > If the underlying data source being queried can provide thousands of (small) > records per second, then only reading them at 100 per RPC call will be > unnecessarily slow. > On the other hand, if the underlying data source is only providing records at > a rate of 1 per second, then it will take 100 seconds for each RPC call to > return, which can lead to timeouts (particularly if Avatica server is sitting > behind a proxy that has a strict request timeout). > The main factors to take into account when finding an ideal size of frame to > return for each RPC call are: > * make the frames small enough that they don't overload either Avatica server > or the client with overly large amounts of data at one time > * make the frames large enough so that the percentage of total query time > that is spent only on RPC overhead is minimized > The general idea of this ticket is to add a pluggable "frame size limiting" > functionality so that frame size limiting can be done based on the number of > rows, number of bytes, amount of time spent building a frame, or any other > property or combination of properties. > Note that CALCITE-2322 contains some work to allow configuring the size of a > single frame on a Connection or Statement (via the {{setFetchSize}} method), > although it's not yet merged in. That ticket would also be useful, and does > not conflict with the general intent of this ticket. -- This message was sent by Atlassian Jira (v8.3.2#803003)