[
https://issues.apache.org/jira/browse/HIVE-6500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-6500:
-----------------------------------
Status: Patch Available (was: Open)
> Stats collection via filesystem
> -------------------------------
>
> Key: HIVE-6500
> URL: https://issues.apache.org/jira/browse/HIVE-6500
> Project: Hive
> Issue Type: New Feature
> Components: Statistics
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Attachments: HIVE-6500.2.patch, HIVE-6500.3.patch, HIVE-6500.patch
>
>
> Recently, support for stats gathering via counter was [added |
> https://issues.apache.org/jira/browse/HIVE-4632] Although, its useful it has
> following issues:
> * [Length of counter group name is limited |
> https://github.com/apache/hadoop-common/blob/branch-2.3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java?source=c#L340]
> * [Length of counter name is limited |
> https://github.com/apache/hadoop-common/blob/branch-2.3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java?source=c#L337]
> * [Number of distinct counter groups are limited |
> https://github.com/apache/hadoop-common/blob/branch-2.3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java?source=c#L343]
> * [Number of distinct counters are limited |
> https://github.com/apache/hadoop-common/blob/branch-2.3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java?source=c#L334]
> Although, these limits are configurable, but setting them to higher value
> implies increased memory load on AM and job history server.
> Now, whether these limits makes sense or not is [debatable |
> https://issues.apache.org/jira/browse/MAPREDUCE-5680] it is desirable that
> Hive doesn't make use of counters features of framework so that it we can
> evolve this feature without relying on support from framework. Filesystem
> based counter collection is a step in that direction.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
