Martin Andersson created SEDONA-64:
--------------------------------------
Summary: Broadcast dedupParams in join judgements to reduce task
deserialization time
Key: SEDONA-64
URL: https://issues.apache.org/jira/browse/SEDONA-64
Project: Apache Sedona
Issue Type: Improvement
Reporter: Martin Andersson
Join judgements requires large variables (dedupParams) that Spark will have to
deserialize for every task.
For very large partition counts this adds a significant overhead. For jobs withÂ
sedona.join.numpartition=20000 (34k actual tasks) task deserialization time
takes on average 200 ms.
By broadcasting dedupParams we've been able to reduce task deserialization time
to 11 ms. Total job execution time is reduces by 20%.
https://spark.apache.org/docs/latest/rdd-programming-guide.html#broadcast-variables
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
