[
https://issues.apache.org/jira/browse/HIVE-17638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janaki Lahorani reassigned HIVE-17638:
--------------------------------------
Assignee: Janaki Lahorani
> SparkDynamicPartitionPruner loads all partition metadata into memory
> --------------------------------------------------------------------
>
> Key: HIVE-17638
> URL: https://issues.apache.org/jira/browse/HIVE-17638
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Janaki Lahorani
>
> The {{SparkDynamicPartitionPruner}} first loads the contents of each
> partition pruning file into memory, and then prunes all the partitions from
> the {{MapWork}}. This can cause increased memory pressure on the HoS Remote
> Driver because it requires loading all the partition metadata into memory. It
> would be more efficient if pruning of partitions was done while scanning the
> files, so that all the partition metadata doesn't need to be buffered in
> memory.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
