[jira] [Commented] (HIVE-20056) SparkPartitionPruner shouldn't be triggered by Spark tasks

[Rui Li (JIRA)]

    [ 
https://issues.apache.org/jira/browse/HIVE-20056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550173#comment-16550173
 ] 

Rui Li commented on HIVE-20056:
-------------------------------

Thanks [~stakiar], nice catch. Patch looks good to me.
Nit:
{code}
+    if (work instanceof MapWork) {
+      if (HiveConf.isSparkDPPAny(jobConf)) {
{code}
=> {{if (work instanceof MapWork && HiveConf.isSparkDPPAny(jobConf))}}

{code}
       if (spec == null) {
-        throw new AssertionException("No partition spec found in dynamic 
pruning");
+        throw new IllegalStateException("No partition spec found in dynamic 
pruning");
       }
{code}
Just use Preconditions?

> SparkPartitionPruner shouldn't be triggered by Spark tasks
> ----------------------------------------------------------
>
>                 Key: HIVE-20056
>                 URL: https://issues.apache.org/jira/browse/HIVE-20056
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Sahil Takiar
>            Assignee: Sahil Takiar
>            Priority: Major
>         Attachments: HIVE-20056.1.patch
>
>
> It looks like {{SparkDynamicPartitionPruner}} is being called by every Spark 
> task because it gets created whenever {{getRecordReader}} is called on the 
> associated {{InputFormat}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)