zhoukang created SPARK-23664:
--------------------------------

             Summary: Add interface to collect query result through file 
iterator
                 Key: SPARK-23664
                 URL: https://issues.apache.org/jira/browse/SPARK-23664
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 2.3.0, 2.1.1
            Reporter: zhoukang


Currently, we use spark sql through jdbc.
Result may cost much memory since we collect result and cached in memory for 
performance consideration.
However,we can also add an API to collect result through file iterator(like 
parquet file iterator),we can avoid OOM of thriftserver for big query.
Like below:

{code:java}
result.toLocalIteratorThroughFile.asScala
{code}
I will work on this if make sense!
And in our internal cluster we have used this API for about a year.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to