[jira] [Commented] (HIVE-18378) Explain plan should show if a Map/Reduce Work is being cached
[ https://issues.apache.org/jira/browse/HIVE-18378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464430#comment-16464430 ] Sahil Takiar commented on HIVE-18378: - Hey [~kellyzly] are you still working on this? > Explain plan should show if a Map/Reduce Work is being cached > - > > Key: HIVE-18378 > URL: https://issues.apache.org/jira/browse/HIVE-18378 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: liyunzhang >Priority: Major > Attachments: HIVE-18378.1.patch > > > It would be nice if the explain plan showed what {{MapWork}} / {{ReduceWork}} > objects are being cached by Spark. > The {{CombineEquivalentWorkResolver}} is the only code that triggers Spark > cache-ing, so we should be able to modify it so that it displays if a work > object will be cached or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18378) Explain plan should show if a Map/Reduce Work is being cached
[ https://issues.apache.org/jira/browse/HIVE-18378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355893#comment-16355893 ] Sahil Takiar commented on HIVE-18378: - [~kellyzly] yes feel free to take this up. I spent some time working on this a few weeks ago, but hit a few issues. I attached a patch of what I had done so far. Feel free to use that the basis of this work (or not). The main issue I was hitting was that the {{CombineEquivalentWorkResolver}} was being triggered in a lot of different queries. More places than I thought it should be. > Explain plan should show if a Map/Reduce Work is being cached > - > > Key: HIVE-18378 > URL: https://issues.apache.org/jira/browse/HIVE-18378 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: liyunzhang >Priority: Major > Attachments: HIVE-18378.1.patch > > > It would be nice if the explain plan showed what {{MapWork}} / {{ReduceWork}} > objects are being cached by Spark. > The {{CombineEquivalentWorkResolver}} is the only code that triggers Spark > cache-ing, so we should be able to modify it so that it displays if a work > object will be cached or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18378) Explain plan should show if a Map/Reduce Work is being cached
[ https://issues.apache.org/jira/browse/HIVE-18378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340647#comment-16340647 ] liyunzhang commented on HIVE-18378: --- [~stakiar]: can you assign thisĀ jira to me ? I guess this feature is need by HIVE-17486 andĀ once HIVE-17486 is resolved, it is better to show whether the work is cached in explain. > Explain plan should show if a Map/Reduce Work is being cached > - > > Key: HIVE-18378 > URL: https://issues.apache.org/jira/browse/HIVE-18378 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: liyunzhang >Priority: Major > > It would be nice if the explain plan showed what {{MapWork}} / {{ReduceWork}} > objects are being cached by Spark. > The {{CombineEquivalentWorkResolver}} is the only code that triggers Spark > cache-ing, so we should be able to modify it so that it displays if a work > object will be cached or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18378) Explain plan should show if a Map/Reduce Work is being cached
[ https://issues.apache.org/jira/browse/HIVE-18378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312093#comment-16312093 ] Sahil Takiar commented on HIVE-18378: - The {{CombineEquivalentWorkResolver}} should also print something in the log everytime it decides to combine two work objets. Right now it doesn't print anything at the INFO level. > Explain plan should show if a Map/Reduce Work is being cached > - > > Key: HIVE-18378 > URL: https://issues.apache.org/jira/browse/HIVE-18378 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar > > It would be nice if the explain plan showed what {{MapWork}} / {{ReduceWork}} > objects are being cached by Spark. > The {{CombineEquivalentWorkResolver}} is the only code that triggers Spark > cache-ing, so we should be able to modify it so that it displays if a work > object will be cached or not. -- This message was sent by Atlassian JIRA (v6.4.14#64029)