[
https://issues.apache.org/jira/browse/SPARK-22189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-22189.
-------------------------------
Resolution: Invalid
Questions should go to the mailing list, please.
> Number of jobs created while querying partitioned table in hive using spark
> ---------------------------------------------------------------------------
>
> Key: SPARK-22189
> URL: https://issues.apache.org/jira/browse/SPARK-22189
> Project: Spark
> Issue Type: Question
> Components: SQL
> Affects Versions: 1.6.0
> Reporter: Astha Arya
>
> I am using Spark SQL
> Spark version - 1.6.0
> Hive 1.1.0-cdh5.9.0
> When I run hiveContext.sql, creates 2 another job for my case i.e. 3 jobs in
> total for querying hive for a partitioned table. Whereas when i run the same
> query on hive using spark as execution engine, it makes only one job.
> Also, the driver logs show that it lists all the partitions which most likely
> shouldnt happen because it slows down my execution.
> Is this a bug? Is there any way to reduce the number of jobs and also not
> list all the partitions each time I query the same table ?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
