[
https://issues.apache.org/jira/browse/HIVE-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198571#comment-15198571
]
Rui Li commented on HIVE-13293:
-------------------------------
My understanding is that to do the sampling, we need to compute the RDD, which
can be a big overhead for complicated queries. Therefore the optimization only
works for queries where order by is dominant. The best use case should be just
ordering a big table. For complicated queries, the re-computation of RDD may
eventually hurt the performance.
MR doesn't have this problem because MR launches a separate job to do the
ordering, and the data to be sampled is already on HDFS.
I think one possible solution is that we can break the spark work at parallel
order by, i.e. just as MR, we compute everything to be sorted, and then launch
a separate spark job to just do the ordering. I can do a PoC to see how this
works.
[~xuefuz] what do you think?
> Query occurs performance degradation after enabling parallel order by for
> Hive on sprak
> ---------------------------------------------------------------------------------------
>
> Key: HIVE-13293
> URL: https://issues.apache.org/jira/browse/HIVE-13293
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Affects Versions: 2.0.0
> Reporter: Lifeng Wang
>
> I use TPCx-BB to do some performance test on Hive on Spark engine. And found
> query 10 has performance degradation when enabling parallel order by.
> It seems that sampling cost much time before running the real query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
