SongXun created SPARK-24900:
-------------------------------
Summary: speed up sort when the dataset is small
Key: SPARK-24900
URL: https://issues.apache.org/jira/browse/SPARK-24900
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 2.3.1
Reporter: SongXun
when running the sql like 'select * from order where order_status = 4 order by
order_id', the pysical plan is:
!https://ws1.sinaimg.cn/large/006tNc79ly1ftl01gtso1j31kw0ag43q.jpg!
the Exchange rangepartitioning has two steps:
1. sample the rdd and get the RangePartitioner which has a rangeBounds
!https://ws1.sinaimg.cn/large/006tNc79ly1ftl0a6ziytj30le0su0vw.jpg!
2. get the rddWithPartitionIds depending on the rangeBounds, and do the shuffle
!https://ws1.sinaimg.cn/large/006tNc79ly1ftl0bapn6mj30l40ke769.jpg!
!https://ws1.sinaimg.cn/large/006tNc79ly1ftl0brnj3jj30kq0mu40q.jpg!
The filescan and filter will be executed twice, it may take a long time. If
the final dataset is small, and the sample data covers all the data, there is
no need to do so.
!https://ws3.sinaimg.cn/large/006tNc79ly1ftl3u0091wj31600a8wh1.jpg!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
