zhoukang created SPARK-23664:
--------------------------------
Summary: Add interface to collect query result through file
iterator
Key: SPARK-23664
URL: https://issues.apache.org/jira/browse/SPARK-23664
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 2.3.0, 2.1.1
Reporter: zhoukang
Currently, we use spark sql through jdbc.
Result may cost much memory since we collect result and cached in memory for
performance consideration.
However,we can also add an API to collect result through file iterator(like
parquet file iterator),we can avoid OOM of thriftserver for big query.
Like below:
{code:java}
result.toLocalIteratorThroughFile.asScala
{code}
I will work on this if make sense!
And in our internal cluster we have used this API for about a year.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
