Varun Saxena commented on YARN-4053:

[~sjlee0], [~djp], [~vrushalic] and others, kindly review.
In this patch I simply assume that we will handle all metrics as longs. 
When AggregationOperation is SUM(which is added as a cell tag/attribute), on 
the coprocessor side I assume this is for metrics because this is what it is 
meant for as of now. 
If we change this for something else, we can change the tag to indicate if this 
cell would contain a long too.

Moreover, I have added a simple flag in ColumnHelper to indicate value has to 
be encoded and decoded as a long. Once we add support for other metrics, maybe 
TimelineMetric should indicate data type of metric but that is not being done 
for now so a flag(the approach adopted in patch) should do, as of now.

Now there were a few other things which have to be handled as part of this 
patch but we have not yet reached consensus on or discussed them.
# We need to decide how to indicate if a metric is to be aggregated or not. 
Currently this kept part of column qualifier in YARN-3816. We can continue with 
that I guess. As MR job run for offline aggregation would need this info as 
# Decide which metric is a TIME_SERIES and which one is SINGLE_VALUE(get only 
the latest value). Should we use tags for it and attach coprocessor with every 
table storing metrics ? Performance implications if that's done ?

> Change the way metric values are stored in HBase Storage
> --------------------------------------------------------
>                 Key: YARN-4053
>                 URL: https://issues.apache.org/jira/browse/YARN-4053
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: YARN-2928
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>         Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.

This message was sent by Atlassian JIRA

Reply via email to