[
https://issues.apache.org/jira/browse/SPARK-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16564885#comment-16564885
]
Izek Greenfield commented on SPARK-13346:
-----------------------------------------
What the status of that? we face this issue too!
> Using DataFrames iteratively leads to slow query planning
> ---------------------------------------------------------
>
> Key: SPARK-13346
> URL: https://issues.apache.org/jira/browse/SPARK-13346
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Joseph K. Bradley
> Priority: Major
>
> I have an iterative algorithm based on DataFrames, and the query plan grows
> very quickly with each iteration. Caching the current DataFrame at the end
> of an iteration does not fix the problem. However, converting the DataFrame
> to an RDD and back at the end of each iteration does fix the problem.
> Printing the query plans shows that the plan explodes quickly (10 lines, to
> several hundred lines, to several thousand lines, ...) with successive
> iterations.
> The desired behavior is for the analyzer to recognize that a big chunk of the
> query plan does not need to be computed since it is already cached. The
> computation on each iteration should be the same.
> If useful, I can push (complex) code to reproduce the issue. But it should
> be simple to see if you create an iterative algorithm which produces a new
> DataFrame from an old one on each iteration.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
