[ https://issues.apache.org/jira/browse/SPARK-24900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-24900: ------------------------------------ Assignee: (was: Apache Spark) > speed up sort when the dataset is small > --------------------------------------- > > Key: SPARK-24900 > URL: https://issues.apache.org/jira/browse/SPARK-24900 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 2.3.1 > Reporter: SongXun > Priority: Minor > Labels: pull-request-available > Original Estimate: 1h > Remaining Estimate: 1h > > when running the sql like 'select * from order where order_status = 4 order > by order_id', the pysical plan is: > !https://ws1.sinaimg.cn/large/006tNc79ly1ftl01gtso1j31kw0ag43q.jpg! > the Exchange rangepartitioning has two steps: > 1. sample the rdd and get the RangePartitioner which has a rangeBounds > !https://ws1.sinaimg.cn/large/006tNc79ly1ftl0a6ziytj30le0su0vw.jpg! > 2. get the rddWithPartitionIds depending on the rangeBounds, and do the > shuffle > !https://ws1.sinaimg.cn/large/006tNc79ly1ftl0bapn6mj30l40ke769.jpg! > !https://ws1.sinaimg.cn/large/006tNc79ly1ftl0brnj3jj30kq0mu40q.jpg! > The filescan and filter will be executed twice, it may take a long time. If > the final dataset is small, and the sample data covers all the data, there is > no need to do so. > !https://ws3.sinaimg.cn/large/006tNc79ly1ftl3u0091wj31600a8wh1.jpg! -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org