[
https://issues.apache.org/jira/browse/HIVE-8574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14226651#comment-14226651
]
Brock Noland commented on HIVE-8574:
------------------------------------
bq. So unless we're worried about a single job creating so many tasks that it
will run the driver out of memory with all the metrics data, this shouldn't
really be an issue.
Any idea how much memory would be consumed for say 100K tasks?
> Enhance metrics gathering in Spark Client [Spark Branch]
> --------------------------------------------------------
>
> Key: HIVE-8574
> URL: https://issues.apache.org/jira/browse/HIVE-8574
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Marcelo Vanzin
> Assignee: Marcelo Vanzin
>
> The current implementation of metrics gathering in the Spark client is a
> little hacky. First, it's awkward to use (and the implementation is also
> pretty ugly). Second, it will just collect metrics indefinitely, so in the
> long term it turns into a huge memory leak.
> We need a simplified interface and some mechanism for disposing of old
> metrics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
