[
https://issues.apache.org/jira/browse/HIVE-17486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyunzhang updated HIVE-17486:
------------------------------
Attachment: HIVE-17486.2.patch
[~lirui] and [~xuefuz]: I have updated HIVE-17486.2.patch and designed again
because a similar case like DS/query28.sql is run successfully.
This proved that current design(M-M-R) can use RDD cache to reduce the table
scan.
In the latest design doc, there is a simple case. Currently, although i have
added the simple case(spark_optimize_shared_work.q), there is some exception
when running qtest in my local env. Once fix the problem, will trigger Hive QA
to test.
> Enable SharedWorkOptimizer in tez on HOS
> ----------------------------------------
>
> Key: HIVE-17486
> URL: https://issues.apache.org/jira/browse/HIVE-17486
> Project: Hive
> Issue Type: Bug
> Reporter: liyunzhang
> Assignee: liyunzhang
> Attachments: HIVE-17486.1.patch, HIVE-17486.2.patch,
> explain.28.share.false, explain.28.share.true, scanshare.after.svg,
> scanshare.before.svg
>
>
> in HIVE-16602, Implement shared scans with Tez.
> Given a query plan, the goal is to identify scans on input tables that can be
> merged so the data is read only once. Optimization will be carried out at the
> physical level. In Hive on Spark, it caches the result of spark work if the
> spark work is used by more than 1 child spark work. After sharedWorkOptimizer
> is enabled in physical plan in HoS, the identical table scans are merged to 1
> table scan. This result of table scan will be used by more 1 child spark
> work. Thus we need not do the same computation because of cache mechanism.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
