Sahil Takiar created IMPALA-8925:
------------------------------------
Summary: Consider replacing ClientRequestState ResultCache with
result spooling
Key: IMPALA-8925
URL: https://issues.apache.org/jira/browse/IMPALA-8925
Project: IMPALA
Issue Type: Improvement
Components: Backend
Reporter: Sahil Takiar
The {{ClientRequestState}} maintains an internal results cache (which is really
just a {{QueryResultSet}}) in order to provide support for the
{{TFetchOrientation.FETCH_FIRST}} fetch orientation (used by Hue - see
[https://github.com/apache/impala/commit/6b769d011d2016a73483f63b311e108d17d9a083]).
The cache itself has some limitations:
* It caches all results in a {{QueryResultSet}} with limited admission control
integration
* It has a max size, if the size is exceeded the cache is emptied
* It cannot spill to disk
Result spooling could potentially replace the query result cache and provide a
few benefits; it should be able to fit more rows since it can spill to disk.
The memory is better tracked as well since it integrates with both admitted and
reserved memory. Hue currently sets the max result set fetch size to
[https://github.com/cloudera/hue/blob/master/apps/impala/src/impala/impala_flags.py#L61],
would be good to check how well that value works for Hue users so we can
decide if replacing the current result cache with result spooling makes sense.
This would require some changes to result spooling as well, currently it
discards rows whenever it reads them from the underlying
{{BufferedTupleStream}}. It would need the ability to reset the read cursor,
which would require some changes to the {{PlanRootSink}} interface as well.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
