[
https://issues.apache.org/jira/browse/HIVE-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14550283#comment-14550283
]
Rui Li commented on HIVE-10458:
-------------------------------
Hi [~xuefuz], we won't do double sample for approach a1. Because the
{{TotalOrderPartitioner}} is MR-specific.
One interesting thing I found is the qtest {{parallel_orderby.q}}. As I
mentioned above, when sorted data is stored in multiple files, we have to read
these files in a proper order to maintain the global sort. Seems when
retrieving the results in FetchOperator, we rely on InputFormat::getSplits
which is related to the underlying FileSystem and doesn't guarantee an order.
So if I run {{parallel_orderby}} with local-cluster mode (TestSparkCliDriver),
the FS used is LocalFileSystem and it doesn't produce a correct result (in fact
we do produce the correct results but we don't read it in a proper way).
However if I run {{parallel_orderby}} with yarn mode
(TestMiniSparkOnYarnCliDriver), the FS used is DistributedFileSystem and the
result is correct. I also tried sorting the splits in FetchOperator and then
both modes work fine.
Maybe we should verify and fix this in a separate JIRA. What do you think?
> Enable parallel order by for spark [Spark Branch]
> -------------------------------------------------
>
> Key: HIVE-10458
> URL: https://issues.apache.org/jira/browse/HIVE-10458
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Rui Li
> Assignee: Rui Li
> Attachments: HIVE-10458.1-spark.patch, HIVE-10458.2-spark.patch,
> HIVE-10458.3-spark.patch
>
>
> We don't have to force reducer# to 1 as spark supports parallel sorting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
