Improve job counter wire protocol with incremental schema and delta values
--------------------------------------------------------------------------
Key: MAPREDUCE-2025
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2025
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client, jobtracker, task
Affects Versions: 0.20.2
Reporter: Luke Lu
Assignee: Luke Lu
Fix For: 0.22.0
The current job counter protocol is quite verbose (string key and vlong value
pairs) especially when keys are long. By using an incremental schema (key to
key id mapping) and data (vint key id to vlong value delta) protocol, we can
significantly lower the counter overhead and allow us to up (from the current 1
update/minute) the counter update frequency. This jira tracks the user counter
(for apps or tools (pig etc.) built on top of mr framework) improvement.
Framework (mr/hdfs etc) counter improvement is tracked by MAPREDUCE-901 (a more
common case with more optimization opportunities, as the schema is fixed.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
