[ https://issues.apache.org/jira/browse/LIVY-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934489#comment-16934489 ]
Marco Gaido commented on LIVY-667: ---------------------------------- [~runzhiwang] I'd argue that if the data which is present in a single partition doesn't fit in the driver, you just need to increase the memory on the driver... > Support query a lot of data. > ---------------------------- > > Key: LIVY-667 > URL: https://issues.apache.org/jira/browse/LIVY-667 > Project: Livy > Issue Type: Bug > Components: Thriftserver > Affects Versions: 0.6.0 > Reporter: runzhiwang > Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > When enable livy.server.thrift.incrementalCollect, thrift use toLocalIterator > to load one partition at each time instead of the whole rdd to avoid > OutOfMemory. However, if the largest partition is too big, the OutOfMemory > still occurs. -- This message was sent by Atlassian Jira (v8.3.4#803005)