[
https://issues.apache.org/jira/browse/SPARK-35787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17366095#comment-17366095
]
Hyukjin Kwon commented on SPARK-35787:
--------------------------------------
Questions should go to mailing list. Let's discuss first before filing it as an
issue.
> Does anyone has performance issue after upgrade from 3.0 to 3.1?
> ----------------------------------------------------------------
>
> Key: SPARK-35787
> URL: https://issues.apache.org/jira/browse/SPARK-35787
> Project: Spark
> Issue Type: Question
> Components: Spark Core
> Affects Versions: 3.1.2
> Reporter: Vidmantas Drasutis
> Priority: Major
> Attachments: Execution_plan_difference.png,
> spark_3.0_execution_plan_details_fast.txt,
> spark_3.1_execution_plan_details_slow.txt, spark_job_info_1.png,
> spark_job_info_2.png
>
>
> Hello.
>
> We had using spark 3.0.2 and query was executed in ~100 seconds.
> After we upgraded Spark to 3.1.1 (tried also 3.1.2 - same, slow performance)
> - our query execution time started taking ~260 seconds it is huge increase
> 250-300 % of execution time increase.
>
> We tried quite simple query.
> In query we using UDF (*org.apache.spark.sql.functions*)
> ) - which explodes data and do polygon hit test. Nothing changed in our code
> from query perspective.
> It is 1 VM box cluster
>
> Maybe anyone faced similar issue?
> Attached some details from spark dashboard.
>
> *Looks like it is UDF related slowdown. As queries which does not use UDF`s
> performance is same and which uses UDFs - starting from 3.1 performance
> decreased.*
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
