[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2016-07-10 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369735#comment-15369735
 ] 

Hudson commented on YARN-4053:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #10074 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10074/])
YARN-4053. Change the way metric values are stored in HBase Storage (sjlee: rev 
51254a6b5133c8abfec4b7d2ac9477d112b3ccfa)
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/application/ApplicationColumnPrefix.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/ColumnHelper.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/package-info.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/entity/EntityColumn.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/TestHBaseStorageFlowRun.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/entity/EntityColumnPrefix.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/FlowRunEntityReader.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/ValueConverter.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunColumn.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowScanner.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/NumericValueConverter.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestHBaseTimelineStorage.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/GenericConverter.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunColumnPrefix.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/LongConverter.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/TimelineStorageUtils.java


> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Fix For: YARN-2928
>
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, 
> YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch, 
> YARN-4053-feature-YARN-2928.06.patch, YARN-4053-feature-YARN-2928.07.patch, 
> YARN-4053-feature-YARN-2928.08.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-20 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15018447#comment-15018447
 ] 

Varun Saxena commented on YARN-4053:


Thanks [~sjlee0] for the review and commit. And [~jrottinghuis] and 
[~vrushalic] for the reviews.

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Fix For: YARN-2928
>
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, 
> YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch, 
> YARN-4053-feature-YARN-2928.06.patch, YARN-4053-feature-YARN-2928.07.patch, 
> YARN-4053-feature-YARN-2928.08.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-19 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014815#comment-15014815
 ] 

Hadoop QA commented on YARN-4053:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 8s 
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
27s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s 
{color} | {color:green} feature-YARN-2928 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s 
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
13s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s 
{color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
23s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
40s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 16s 
{color} | {color:red} hadoop-yarn-server-timelineservice in feature-YARN-2928 
failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s 
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 19s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s 
{color} | {color:red} Patch generated 1 new checkstyle issues in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
 (total was 42, now 38). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
42s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 15s 
{color} | {color:red} hadoop-yarn-server-timelineservice in the patch failed 
with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 5s 
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed 
with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 55s 
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed 
with JDK v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
25s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 17s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:date2015-11-19 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12773366/YARN-4053-feature-YARN-2928.07.patch
 |
| JIRA Issue | YARN-4053 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  

[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-19 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014822#comment-15014822
 ] 

Sangjin Lee commented on YARN-4053:
---

Could you address that new checkstyle violation? Then I think it's good to go. 
Thanks!

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, 
> YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch, 
> YARN-4053-feature-YARN-2928.06.patch, YARN-4053-feature-YARN-2928.07.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-19 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014487#comment-15014487
 ] 

Hadoop QA commented on YARN-4053:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s 
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m 
12s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s 
{color} | {color:green} feature-YARN-2928 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s 
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
15s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s 
{color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
21s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
47s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 28s 
{color} | {color:red} hadoop-yarn-server-timelineservice in feature-YARN-2928 
failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s 
{color} | {color:red} Patch generated 6 new checkstyle issues in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
 (total was 43, now 45). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
56s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 27s 
{color} | {color:red} hadoop-yarn-server-timelineservice in the patch failed 
with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 25s 
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed 
with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 56s 
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed 
with JDK v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
29s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 56s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:date2015-11-19 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12773338/YARN-4053-feature-YARN-2928.06.patch
 |
| JIRA Issue | YARN-4053 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  

[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-19 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014509#comment-15014509
 ] 

Sangjin Lee commented on YARN-4053:
---

Also, please look at the checkstyle violations. There are new ones that are 
introduced by this patch.

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, 
> YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch, 
> YARN-4053-feature-YARN-2928.06.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-19 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014432#comment-15014432
 ] 

Varun Saxena commented on YARN-4053:


This patch removes ValueConverterImpl and uses generics as suggested by 
[~sjlee0].
Haven't added varargs in add method as varargs don't go well with generics and 
can lead to class cast exceptions. Maybe concrete class types can be used as 
varargs but that would mean mixing generic params with concrete type 
param(Number...) in the same method. So haven't added them

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, 
> YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch, 
> YARN-4053-feature-YARN-2928.06.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-19 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014673#comment-15014673
 ] 

Varun Saxena commented on YARN-4053:


This patch is similar to version 5 of the patch. No longer using generics. 
Removed ValueConverterImpl.
And changed add method to {{Number add(Number num1, Number num2, 
Number...numbers)}} 

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, 
> YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch, 
> YARN-4053-feature-YARN-2928.06.patch, YARN-4053-feature-YARN-2928.07.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-19 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014497#comment-15014497
 ] 

Sangjin Lee commented on YARN-4053:
---

As mentioned above the generic version is good only if {{add()}} and 
{{compare()}} would operate only on return values of {{decodeValue()}}. I'm not 
sure if that's always a reliable assumption although it happens to be true 
right now. [~jrottinghuis]?

Can we go back to non-generic version for the ValueConverter hierarchy? Sorry 
for going back and forth on this one!

Just one more minor nit. In {{LongConverter}}, {{decodeValue()}} can simply 
return {{Bytes.toLong()}} without using {{Long.valueOf()}}. The same goes for 
{{add()}}. Autoboxing will take care of that for you.


> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, 
> YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch, 
> YARN-4053-feature-YARN-2928.06.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-19 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014518#comment-15014518
 ] 

Varun Saxena commented on YARN-4053:


Just to clarify, by mixing I meant I was thinking of a method like {{T add(T 
num1, T num2, T...numbers)}} as minimum 2 numbers are required to add. Anyways 
if we change the method to original suggestion i.e. {{T 
add(Number...numbers)}}, it would work. But then we are not really utilizing 
generics here, atleast for add method. Let me know your thoughts on this.

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, 
> YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch, 
> YARN-4053-feature-YARN-2928.06.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-19 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014525#comment-15014525
 ] 

Varun Saxena commented on YARN-4053:


Ok.

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, 
> YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch, 
> YARN-4053-feature-YARN-2928.06.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-19 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15015332#comment-15015332
 ] 

Hadoop QA commented on YARN-4053:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s 
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
44s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s 
{color} | {color:green} feature-YARN-2928 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s 
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
16s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s 
{color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
24s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
54s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 23s 
{color} | {color:red} hadoop-yarn-server-timelineservice in feature-YARN-2928 
failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s 
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 31s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
51s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 20s 
{color} | {color:red} hadoop-yarn-server-timelineservice in the patch failed 
with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 46s 
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed 
with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 46s 
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed 
with JDK v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
29s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 3s {color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:date2015-11-20 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12773453/YARN-4053-feature-YARN-2928.08.patch
 |
| JIRA Issue | YARN-4053 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 82b95b6d1a2e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 

[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-18 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15011590#comment-15011590
 ] 

Hadoop QA commented on YARN-4053:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 7s 
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
25s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s 
{color} | {color:green} feature-YARN-2928 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s 
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
15s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 32s 
{color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
24s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
48s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 20s 
{color} | {color:red} hadoop-yarn-server-timelineservice in feature-YARN-2928 
failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s 
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
50s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 18s 
{color} | {color:red} hadoop-yarn-server-timelineservice in the patch failed 
with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s 
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 23s 
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed 
with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 26s 
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed 
with JDK v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
31s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 5s {color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:date2015-11-18 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12773033/YARN-4053-feature-YARN-2928.05.patch
 |
| JIRA Issue | YARN-4053 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 59b383bea07a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 

[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-18 Thread Joep Rottinghuis (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15011789#comment-15011789
 ] 

Joep Rottinghuis commented on YARN-4053:


Thanks [~varun_saxena] patch looks good to me. Thanks for the updates!
[~sjlee0] and [~vrushalic] said they'll look at this patch by end of day today 
as well.

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, 
> YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-18 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15012508#comment-15012508
 ] 

Sangjin Lee commented on YARN-4053:
---

Sorry, I should have double checked. It would be more like

{code}
public interface NumberValueConverter extends ValueConverter,
Comparator {
  T add(T num1, T num2);
}
{code}

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, 
> YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-18 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15012620#comment-15012620
 ] 

Sangjin Lee commented on YARN-4053:
---

Yes, I spoke with [~jrottinghuis] regarding the stronger type safety 
suggestion. I'm now more or less comfortable with the current version. The 
additional type safety you get is probably minimal, and can cause bit more 
headache as a side effect as Joep mentioned.

Just for the record, I tinkered with this a little bit, and there seems to be a 
way to do this safely. The assumption is {{add()}} and {{compare()}} are 
invoked only on the return values of {{decodeValue()}}. That is currently the 
case, but it might not always be true. In any case, if that were always true, 
we could do

{code}
public interface ValueConverter {
  byte[] encodeValue(Object value) throws IOException; // this value is still 
object
  T decodeValue(byte[] bytes) throws IOException;
}

public interface NumericValueConverter
extends ValueConverter, Comparator {
  T add(T num1, T num2);
}

public class LongConverter implements NumericValueConverter {
  ...
}
{code}

This is basically tying the loop from {{decodeValue()}} and {{add()}} and 
{{compare()}}. Again, this is a mild suggestion at this point, and it works 
only if the assumption that {{add()}} and {{compare()}} work only on the result 
of {{decodeValue()}} is true.

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, 
> YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-18 Thread Joep Rottinghuis (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15012595#comment-15012595
 ] 

Joep Rottinghuis commented on YARN-4053:


[~sjlee0] the ValueConverterImpl is a remnant from earlier implementation where 
the enum had anonymous inner classes in the enum values. That is no longer 
needed now and we can probably eliminate that indeed.
 
If we go with the tightened type interface then we would be able to add only 
Longs to Longs etc. That by itself is probably not too bad, because the column 
would be defined as one or the other. However, in FlowScanner.processSummation 
this will be an issue. In that method we don't know what the type for each 
column is, so the sum member is defined as a Number to future proof potentially 
other conversions. This is why we made the conversion pluggable. The add  
method must accept two numbers and emit a number. We could still have the class 
accept any T extends Number and if needed use that T later in the class to 
ensure that number implements T (or else throw an exception) if we want to 
implement the restriction that Vrushali mentions. The add method would still 
accept Number in the signature I think.

This did make me think of another thing: right now we add only two numbers, 
never more than two. If at some point we want to add three or more Numbers we 
would have do to that one at a time. We could change the signature of 
NumericConverter from
{code}
 public Number add(Number num1, Number num2);
{code}
to
{code}
 public Number add(Number... numbers);
{code}
The calling code wouldn't have to change because it passes two numbers, but 
then a possible future use-case could add three or more arguments together.
The implementation would change slightly to check for null and return null, or 
iterate over the list and return the sum of all values (that aren't null). 
That is probably more a matter of which style you like better, because right 
now we don't have this use-case.

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, 
> YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-18 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15012185#comment-15012185
 ] 

Sangjin Lee commented on YARN-4053:
---

Sorry it took me awhile to review the latest patch. It looks good for the most 
part. I have a couple of minor comments.

- The name {{ValueConverterImpl}} threw me off a little because initially I 
thought it was about actual implementations of {{ValueConverter}}. It seems 
more like a {{ValueConverterFactory}}. I think we can go straight to 
{{GenericConverter.getIstance()}} and {{LongConverter.getInstance()}}. The 
factory seems like a little overkill.
- Can we change {{NumericValueConverter}} to use generics; e.g.
{code}
public interface NumericValueConverter extends ValueConverter,
Comparator {

{code}
? This can be even tighter than the current interface.

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, 
> YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-18 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15012244#comment-15012244
 ] 

Sangjin Lee commented on YARN-4053:
---

Is it the same question as what Varun answered earlier?

bq. JAX-RS i.e. the REST API layer will convert an integral value to Integer 
automatically if its less than Integer.MAX_VALUE so I guess we will have to 
handle ints and shorts as well i.e. if its an Integer for instance, we can call 
Integer#longValue to convert it to long.


> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, 
> YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-18 Thread Vrushali C (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15012422#comment-15012422
 ] 

Vrushali C commented on YARN-4053:
--

Alright, [~sjlee0] explained to me that there are writes coming via REST, so it 
is okay to have the isIntegralValue check in there. Thanks [~sjlee0]! 

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, 
> YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-18 Thread Vrushali C (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15012208#comment-15012208
 ] 

Vrushali C commented on YARN-4053:
--

Thanks [~varun_saxena] for the patch! It looks good to me except for the 
following question. 

One point  I recollect was that we wanted to accept only longs while encoding. 
In patch v4 I see LongConverter #encodeValue() accept any integral value and 
encode it as Longs. Am wondering if we should accept only longs? That is, 
modify the isIntegralValue check to isLongValue? cc [~jrottinghuis] 

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, 
> YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-18 Thread Vrushali C (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15012313#comment-15012313
 ] 

Vrushali C commented on YARN-4053:
--

 I thought that was only for client side read queries. Is the REST API layer 
being invoked for writes as well?  So each container makes a rest call to the 
NM when it has to write a metric? I am wondering if that will be very slow?

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, 
> YARN-4053-feature-YARN-2928.04.patch, YARN-4053-feature-YARN-2928.05.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-17 Thread Joep Rottinghuis (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009162#comment-15009162
 ] 

Joep Rottinghuis commented on YARN-4053:


Either throw exception, or document in javadoc that null is considered 0. For 
addition that would be reasonable, but either approach is fine (as long as the 
javadoc explains which approach was taken).
For comparison I suppose only null == null is reasonable. For null compared to 
anything else you can either assume 0 or throw an exception. Not sure that one 
can say that null is bigger or smaller than any other value.

Wrt comparable versus comparator you're correct. We're not comparing the 
ValueConverters themselves, but rather Numbers.

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, 
> YARN-4053-feature-YARN-2928.04.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-17 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009115#comment-15009115
 ] 

Varun Saxena commented on YARN-4053:


bq. public interface NumericValueConverter extends ValueConverter, 
Comparable ?
I guess you mean Comparator. Yes we can have that.

bq. LongConverter.compare and LongConverter.add should probably handle null 
values. 
Throw an exception if either is null ?

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, 
> YARN-4053-feature-YARN-2928.04.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-16 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007358#comment-15007358
 ] 

Hadoop QA commented on YARN-4053:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 6s 
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 
8s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s 
{color} | {color:green} feature-YARN-2928 passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s 
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
12s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s 
{color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
21s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
38s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 17s 
{color} | {color:red} hadoop-yarn-server-timelineservice in feature-YARN-2928 
failed with JDK v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s 
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s 
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 23s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
41s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 15s 
{color} | {color:red} hadoop-yarn-server-timelineservice in the patch failed 
with JDK v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 35s 
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed 
with JDK v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 48s 
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed 
with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
24s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 36s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.1 Server=1.7.1 
Image:test-patch-base-hadoop-date2015-11-16 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12772577/YARN-4053-feature-YARN-2928.04.patch
 |
| JIRA Issue | YARN-4053 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 2f046c7ddf23 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 

[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-16 Thread Joep Rottinghuis (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008041#comment-15008041
 ] 

Joep Rottinghuis commented on YARN-4053:


Looks good [~varun_saxena] this is a nice separation of the conversion and the 
numeric conversion / comparison and general numeric manipulation.

Nit: LongConverter.compare and LongConverter.add should probably handle null 
values. 
Question any reason you don't simply have
public interface NumericValueConverter extends ValueConverter, 
Comparable ?


> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch, 
> YARN-4053-feature-YARN-2928.04.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-13 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003688#comment-15003688
 ] 

Hadoop QA commented on YARN-4053:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s 
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 
10s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s 
{color} | {color:green} feature-YARN-2928 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s 
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
15s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s 
{color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
25s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
43s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 21s 
{color} | {color:red} hadoop-yarn-server-timelineservice in feature-YARN-2928 
failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s 
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 18s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 18s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s 
{color} | {color:red} Patch generated 4 new checkstyle issues in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
 (total was 41, now 45). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
49s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 19s 
{color} | {color:red} hadoop-yarn-server-timelineservice in the patch failed 
with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 43s 
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed 
with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 41s 
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed 
with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
27s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 53s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.0 Server=1.7.0 
Image:test-patch-base-hadoop-date2015-11-13 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12772138/YARN-4053-feature-YARN-2928.03.patch
 |
| JIRA Issue | YARN-4053 |
| Optional Tests |  asflicense  compile  javac  javadoc  

[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-13 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003690#comment-15003690
 ] 

Varun Saxena commented on YARN-4053:


Updating a renamed patch file so that QA build can run.

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-13 Thread Joep Rottinghuis (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15004550#comment-15004550
 ] 

Joep Rottinghuis commented on YARN-4053:


Patch looks good. The converter approach looks nice and clean.

One thing I'm wondering if we need to look up the converter each time in 
FlowScanner line 180 where we do getValueConverter each time in nextInternal.
I'm wondering if we can somehow avoid this. Would it be safe to always assume a 
long converter? What happens if we did end up mixing a long and a non-long 
converter? Perhaps that is not a safe assumption.
Would it be possible to do this once and not each time? I'll chat with 
[~vrushalic] if we can somehow get around iterating over every column for each 
cell. Perhaps we can capture this during the constructor.

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-13 Thread Joep Rottinghuis (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15004842#comment-15004842
 ] 

Joep Rottinghuis commented on YARN-4053:


Now that I'm looking at it again in detail with [~vrushalic] the perf impact is 
probably not quite as bad. I thought for a moment that the iteration had to 
happen for each cell, but it is needed for each column only.

I agree with your point that we should keep it generic and be future proof. If 
we have the converter, then we should just use it. However, in 
FlowScanner.compareCellValues (line 357) you're assuming a long and just 
blindly cast to long. If this happens to be a GenericObjectMapper then that 
will fail.

Perhaps we should do the following: make converter extend comparable and also 
add a method to sum two (or more) byte values in a vararg style. For the long 
implementation that should simply be a regular sum, for the generic object 
mapper it may be a string append, or whatever makes sense. Then we're future 
proof when another style of conversion and summing and comparison can happen 
safely right?

Would that make sense?

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-13 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15004727#comment-15004727
 ] 

Varun Saxena commented on YARN-4053:


bq. Would it be safe to always assume a long converter?
I think while processing it would be safe to assume with current columns and 
converters. The issue comes if somebody changes the converter corresponding to 
column/column prefix in future. And possibly forgets to change here. Arguably 
though its difficult to miss such a change after testing.

The reason this iterating over all columns/column prefixes was added was to 
handle future column additions or change in converter for existing columns.

Getting the converter on every iteration was a concern even for me. That's why 
I am doing it only when column qualifier changes while iterating cells.
Not sure if we can get hold of all possible column qualifiers(and hence the 
converter) for this scan in the constructor.

Another option would be to use cell tags to identify converters.




> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-13 Thread Joep Rottinghuis (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15004887#comment-15004887
 ] 

Joep Rottinghuis commented on YARN-4053:


In further discussion with [~sjlee0] it may actually be that the current 
implementation of the FlowScanner doesn't properly deal with mixed numeric and 
non-numeric columns.
Perhaps that is a separate jira to properly deal with that. I think what we may 
have to do is to ensure that we normally return any column that is non-numeric 
untouched.

Sangjin suggested that perhaps the new methods I mentioned (comparator, and 
sum) make sense only for numeric types. That can be more cleanly implemented as 
a sub-interface. Then the FlowScanner can determine if the returned converter 
is numeric. If so, it can process as is with collapsing values, or else it 
would simply leave the cells untouched. That way we could mix numeric and 
non-numeric columns in one column family and we avoid having to implement a sum 
or a meaningless comparison between unrelated objects.

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-10 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999660#comment-14999660
 ] 

Sangjin Lee commented on YARN-4053:
---

To make progress with this ticket, if you're in line with what Vrushali said 
above, we can focus on implementing the correct long support in this ticket. We 
don't have to worry about the other dimensions (whether to aggregate, or 
single-value v. time series) in here.

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-10 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999780#comment-14999780
 ] 

Varun Saxena commented on YARN-4053:


Ok

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-10 Thread Vrushali C (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999822#comment-14999822
 ] 

Vrushali C commented on YARN-4053:
--

bq. Vrushali, thanks for your comments. I would like to work on this. Let me 
take a stab on this one. Will have the bandwidth. I hope its fine. You can help 
me with the reviews.

Sounds good, let me go through the discussion points you have mentioned and get 
back on this.

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-10 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999841#comment-14999841
 ] 

Varun Saxena commented on YARN-4053:


Moreover, whether to aggregate or not, the proposal is to not have it in a 
column qualifier. So nothing to do here for that. YARN-3816 will have to remove 
code corresponding to it.

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-YARN-2928.03.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-10 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999837#comment-14999837
 ] 

Varun Saxena commented on YARN-4053:


Attached a new patch addressing points above. 
Added a ValueConverter interface and a ValueConverterImpl enum which contains 
GENERIC and LONG implementations.
In FlowScanner, will have to iterate over all the available column prefixes and 
columns to get hold of the right converter.

Haven't addressed TIME_SERIES related point as of now.
Can have it in the next patch once a consensus is reached for the 
implementation..

Functionally speaking, over the last patch I am now storing min start and max 
end time as longs as well.

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch, YARN-4053-YARN-2928.03.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-09 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14997121#comment-14997121
 ] 

Varun Saxena commented on YARN-4053:


Vrushali, thanks for your comments.
I would like to work on this. Let me take a stab on this one. Will have the 
bandwidth.
I hope its fine. You can help me with the reviews.

Coming to the points, 
I agree that flag is not good for extensibility. As I said earlier, flag should 
be fine for now as we have only 2 choices(generic or long) and we can extend 
later. 
But eventually will have to have different handlers for different types. So why 
not do it now. Hence, lets go with proposal above.

Moreover, yes, we need to have proper handling based on data type or conversion 
mechanism in FlowScanner too. As mentioned in an earlier comment, I was 
thinking we can indicate this in attributes. But I guess your proposal sounds 
better. We can identify the column/column prefix in flow scanner as well and 
convert based on the converter attached to it.

bq. it missed one of the places in the current patch for example
Which place ? MIN/MAX handling ?

bq. For single value vs time series, we suggest using a column prefix to 
distinguish them
Do we need to have a differentiation between SINGLE_VALUE and TIME_SERIES if by 
default it will be read as SINGLE_VALUE ? Because we will be storing multiple 
values even for metric of type SINGLE_VALUE. Do you mean on the read side, only 
the latest value of a metric is to be returned if its of type SINGLE_VALUE 
(even if client asks for TIME_SERIES) ? Again the assumption here is that 
client will always send the metric type(SINGLE_VALUE or TIME_SERIES) 
consistently.

bq. For the read path, we can assume it is a single value unless specifically 
specified by the client as a time series (as clients would need to intend to 
read time series explicitly).
We can return TIME_SERIES by indicating something like METRICS_TIME_SERIES as 
fields. If we do so, it will have implications on YARN-3862.
Now the question is whether to return values for multiple timestamps even for 
metric type of SINGLE_VALUE if client asks for it ? What if client wants to see 
values of a gauge(which might be considered as a SINGLE_VALUE) over a period of 
time, for instance. If yes, do we need to even differentiate between the 2 
types ?

bq. We finally concluded that we should start with storing longs only and make 
the code strictly accept longs 
JAX-RS i.e. the REST API layer will convert an integral value to Integer 
automatically if its less than Integer.MAX_VALUE so I guess we will have to 
handle ints and shorts as well i.e. if its an Integer for instance, we can call 
Integer#longValue to convert it to long.

bq. Regarding indicating whether to aggregate or not, we suggest to rely mostly 
on the flow run aggregation. For those use cases that need to access metrics 
off of tables other than the flow run table (e.g. time-based aggregation), we 
need to explore ways to specify this information as input (config, etc.)
I hope Li Lu is fine with this because I remember him saying on YARN-3816 that 
he will be using it for offline aggregation in YARN-3817. I think rows from 
application table are being used in the MR job there. Are you suggesting that 
for offline aggregation, based on config, we aggregate all the application 
metrics(to flow or user) or nothing ?
Or configure a set of metrics to aggregate in some config ?

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-11-06 Thread Vrushali C (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994603#comment-14994603
 ] 

Vrushali C commented on YARN-4053:
--

Thanks [~varun_saxena] for the patch and [~djp] , [~gtCarrera], 
[~Naganarasimha], [~sjlee0] and [~jrottinghuis] for the discussion so far!

[~jrottinghuis] , [~sjlee0] and I had an offline discussion on this yesterday. 
We discussed at length along the following vectors:
- metric datatype: long, double, either or, both?
- metric type storage and retrieval for: single values vs timeseries
- metrics in the context of aggregation: how to indicate whether to aggregate 
or no.
- operations on metrics: sum vs average, min/max

To summarize the discussion:

- Our proposal is to proceed with supporting only longs for now. We went over 
several situations of how to store and query for decimal numbers: as Doubles or 
as numerator/denominator, how to use filters while scanning for such stored 
values,  how would aggregation look at it etc. We thought about which metrics 
are to be stored as Doubles and how the precision might affect aggregation. We 
finally concluded that we should start with storing longs only and make the 
code strictly accept longs (not even ints or shorts).

- For single value vs time series, we suggest using a column prefix to 
distinguish them. For the read path, we can assume it is a single value unless 
specifically specified by the client as a time series (as clients would need to 
intend to read time series explicitly).

- Regarding indicating whether to aggregate or not, we suggest to rely mostly 
on the flow run aggregation. For those use cases that need to access metrics 
off of tables other than the flow run table (e.g. time-based aggregation), we 
need to explore ways to specify this information as input (config, etc.) 

- So, the current patch is along the lines of our proposal of using longs for 
metrics. But we are considering a different approach of creating a "converter" 
type and implementation. For other non metric columns, a "generic" converter 
that uses the GenericObjectMapper can be created and used implicitly. For the 
numeric (long) columns, a long converter would be used explicitly. We also need 
to revisit how it's done in FlowScanner (it missed one of the places in the 
current patch for example). We need to get at the instances of ColumnPrefix and 
ColumnFamily, etc. and use them to get the converter in the flow scanner.

@Varun Would it be fine if I took over this jira to patch it with the above 
points?

thanks
Vrushali

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-10-26 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14974100#comment-14974100
 ] 

Hadoop QA commented on YARN-4053:
-

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch |  16m 45s | Findbugs (version ) appears to 
be broken on YARN-2928. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to 
include 2 new or modified test files. |
| {color:green}+1{color} | javac |   8m 10s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |  10m 35s | There were no new javadoc 
warning messages. |
| {color:red}-1{color} | release audit |   0m 18s | The applied patch generated 
1 release audit warnings. |
| {color:green}+1{color} | checkstyle |   0m 14s | There were no new checkstyle 
issues. |
| {color:green}+1{color} | whitespace |   0m  2s | The patch has no lines that 
end in whitespace. |
| {color:green}+1{color} | install |   1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 40s | The patch built with 
eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   0m 52s | The patch does not introduce 
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests |   2m 55s | Tests passed in 
hadoop-yarn-server-timelineservice. |
| | |  42m 13s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12768689/YARN-4053-YARN-2928.02.patch
 |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-2928 / 3c4e424 |
| Release Audit | 
https://builds.apache.org/job/PreCommit-YARN-Build/9575/artifact/patchprocess/patchReleaseAuditProblems.txt
 |
| hadoop-yarn-server-timelineservice test log | 
https://builds.apache.org/job/PreCommit-YARN-Build/9575/artifact/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/9575/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/9575/console |


This message was automatically generated.

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-10-26 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14974623#comment-14974623
 ] 

Varun Saxena commented on YARN-4053:


[~sjlee0], [~djp], [~vrushalic] and others, kindly review.
In this patch I simply assume that we will handle all metrics as longs. 
When AggregationOperation is SUM(which is added as a cell tag/attribute), on 
the coprocessor side I assume this is for metrics because this is what it is 
meant for as of now. 
If we change this for something else, we can change the tag to indicate if this 
cell would contain a long too.

Moreover, I have added a simple flag in ColumnHelper to indicate value has to 
be encoded and decoded as a long. Once we add support for other metrics, maybe 
TimelineMetric should indicate data type of metric but that is not being done 
for now so a flag(the approach adopted in patch) should do, as of now.

Now there were a few other things which have to be handled as part of this 
patch but we have not yet reached consensus on or discussed them.
# We need to decide how to indicate if a metric is to be aggregated or not. 
Currently this kept part of column qualifier in YARN-3816. We can continue with 
that I guess. As MR job run for offline aggregation would need this info as 
well.
# Decide which metric is a TIME_SERIES and which one is SINGLE_VALUE(get only 
the latest value). Should we use tags for it and attach coprocessor with every 
table storing metrics ? Performance implications if that's done ?

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-4053-YARN-2928.01.patch, 
> YARN-4053-YARN-2928.02.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-09-28 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14933492#comment-14933492
 ] 

Varun Saxena commented on YARN-4053:


I think we need to revive this JIRA and decide its scope.
I will be converting everything to longs as of now. And throw exceptions if its 
a floating point or BigInteger value.

Other than that, we need to decide do we need to differentiate between 
TIME_SERIES and SINGLE_VALUE , indicate to client if a particular metric is an 
aggregated metric or not, etc.
The challenge though is that we cannot use cell tags for anything which needs 
to be sent back to client as a Get/Scan cant get hold of tags on the client 
side.



> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-4053-YARN-2928.01.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-09-02 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14727641#comment-14727641
 ] 

Varun Saxena commented on YARN-4053:


bq. to use cell tag identify metrics value type
In YARN-3816, by metric value type I meant whether its aggregated metric or 
not. I guess that is what you were referring to. Cell tags can be useful that 
way. But identifying the metric's data type by cell tag wont work in our case 
because we have to apply metric filters.

IMHO, I think forcing user to send consistent values is a fair enough 
expectation. 
If user doesnt send, they will get inconsistent results.

If we go by this premise, I think we can go ahead with Longs and Doubles.
Otherwise, we can assume all values as long.

The point which Vrushali raised above regarding floating point precision not 
being that important. Well, as I point out in last Wednesday's meeting and the 
example which [~djp] gives, floating points may not matter for very large 
values but they would matter for metrics whose range is towards the lower 
end(say, 0 to 5).
Now the real question is do we expect such metrics ? 

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-4053-YARN-2928.01.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-09-02 Thread Junping Du (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14727620#comment-14727620
 ] 

Junping Du commented on YARN-4053:
--

Hi [~sjlee0] and all, sorry for coming late for this good discussion. 
bq. we're of the opinion that we can start supporting only longs for now (i.e. 
no floating point types), while we can consider adding a floating point type 
(namely double) to the list of supported types.
If applications want to store value of percentage which are all between 0.0 and 
1.0, what should they do? casting decimal number (0.49 and 0.50) directly to 
integer(long) or multiply 100 and divide 100 later?
I agree that only support Long make things much easier. However, it seems 
cannot satisfy some basic requirements and scenarios. Supporting Long and 
Double for the first step (instead of int, long, float and double) sounds like 
a reasonable compromise though. About solutions, I think [~varun_saxena] bring 
up an option in YARN-3816 to use cell tag identify metrics value type which 
sounds good to me. Do we have any concern on this way? I am quite interested in 
this as we may want to apply more meta info (raw or aggregated, etc.) on 
metrics in future. Thoughts?

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-4053-YARN-2928.01.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-09-02 Thread Joep Rottinghuis (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14727816#comment-14727816
 ] 

Joep Rottinghuis commented on YARN-4053:


Part of the issue is that there are many moving components that all need to 
work together around properly handling longs and doubles. This is aggregation, 
store, read, column value queries, filters, etc.
Tags have challenges pointed out before, and in addition, tags are filtered out 
on read (you will not get all the tag values, unless you do voodoo in a 
coprocessor).

Since we're still figuring out how all these pieces work together in a 
consistent manner, our suggestion is to first make this all work together with 
longs only, and then after that figure out how to add support for double.
Trying to make things work for both will bog us down right now and we'll likely 
not get things working consistently, we simply can't oversee all the 
consequences and impacts on aggregation, querying etc.

> Change the way metric values are stored in HBase Storage
> 
>
> Key: YARN-4053
> URL: https://issues.apache.org/jira/browse/YARN-4053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-4053-YARN-2928.01.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store 
> values in backend HBase storage. This converts everything into a string 
> representation(ASCII/UTF-8 encoded byte array).
> While this is fine in most cases, it does not quite serve our use case for 
> metrics. 
> So we need to decide how are we going to encode and decode metric values and 
> store them in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-27 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14717865#comment-14717865
 ] 

Sangjin Lee commented on YARN-4053:
---

[~vrushalic], [~jrottinghuis], and I discussed supported types a little more, 
and we're of the opinion that we can *start* supporting only longs for now 
(i.e. no floating point types), while we can consider adding a floating point 
type (namely double) to the list of supported types. So for now, how about 
assuming (and enforcing) long as the type of the metric values, and pursue how 
we can add double later if we need it? Thoughts?

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena
 Attachments: YARN-4053-YARN-2928.01.patch


 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-27 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14717863#comment-14717863
 ] 

Sangjin Lee commented on YARN-4053:
---

Thanks [~varun_saxena] for the discussion. As you said, one thing that really 
causes issues is when inconsistent values are used for the same metric. At a 
high level, I think we need to ask these questions:

- How important is it to support this scenario?
- If we don't really support this scenario, then what is the minimally 
acceptable behavior if that were to happen?

The gist of the problem is that one cannot really write/read consistent values 
without knowing the right type of the metric. The user will likely not know 
that either for the write or read path. In the face of this, the main 
difference between approach #1 (encoding it into the value) and approach #2 
(adding it to the column qualifier) is that approach #1 will mix different-type 
values into a single time series (column), and approach #2 will effectively 
create two separate time series (columns). The rest is the fallout.


 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena
 Attachments: YARN-4053-YARN-2928.01.patch


 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-26 Thread Vrushali C (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715192#comment-14715192
 ] 

Vrushali C commented on YARN-4053:
--


The way I see this, it comes down to a basic question of whether we really 
*need* floating point precision in metric values. For instance, cost is a 
metric which could have a decimal value upon calculation. But, in my opinion 
say a cost of 5 dollars versus 5.347891 dollars versus a cost of 5.78913 are 
not that different. A cost of 6.x dollars is different from 5.x.  I believe 
that it does not matter THAT much that cost is 5.347891 or 5.79813.  These are 
hadoop applications, the time duration is rarely going to be exactly consistent 
for the exactly same code. So metrics will usually have a slight fluctuation 
between different runs of the exact same job. 

Storage and querying of Longs is straightforward and clean. No ambiguity in 
serialization. 

Contrasting that with storage of various numerical data types in metrics:
- all the complexity of storing of column prefixes that can tell us which type 
is stored so that serialization to/from hbase can be done correctly.
- the filtering in hbase becomes so much more complicated with all these 
different datatypes.





 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena
 Attachments: YARN-4053-YARN-2928.01.patch


 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-25 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710651#comment-14710651
 ] 

Varun Saxena commented on YARN-4053:


Looking at the issues involved, IMO we should impose restriction on the client 
so that it does not mix longs and doubles. 

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena
 Attachments: YARN-4053-YARN-2928.01.patch


 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-24 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710638#comment-14710638
 ] 

Varun Saxena commented on YARN-4053:


There was a suggestion that we can support only longs. Would supporting only 
longs not cause any impact to potential users of ATS ?
longs however should cover most of the metrics(as of now I can’t think of any 
where decimals would be of great importance).
If we do this, I think TimelineMetric object should be changed to accept only 
java.lang.Long and not java.lang.Number…
Looping [~vinodkv] to get his opinion on this as well.
 
Although, is it unfair to ask client to send values consistently ?
Can’t we document this and enforce this restriction. And if client does not 
comply, it cannot expect consistent results. This can be the contract between 
ATS and its clients.
Major concern here though would be that it won’t be possible to enforce this 
restriction programmatically, neither at the client side nor at the server side.
 
*Possible Solution :*
There is one possible solution though if enforcing this restriction is not 
viable. The real problem in both the solutions would come in applying metric 
filters, if data is inconsistent.
So for this, we can use approach 2(include type in column qualifier) and then 
insert OR filters covering both the column qualifiers for same metric.
 
I will elaborate this with an example.
Let us say we have a metric called JOB_ELAPSED_TIME and client can report both 
integral and floating point values for it(say). With approach 2, we will have 2 
column qualifiers for this metric i.e.  “ JOB_ELAPSED_TIME=L” (for longs) and 
“JOB_ELAPSED_TIME=D” (for doubles).
Now, when a query comes with metric filter value in integer format i.e. 
something like JOB_ELAPSED_TIME  40 can be transformed to corresponding HBase 
filter of the form (“JOB_ELAPSED_TIME=L”  40 OR “JOB_ELAPSED_TIME=D”  40.0).
 i.e. a filter list of the form (“m1”  10 AND “m2”  5 AND “m3”=4) would be 
transformed to ((“m1=L”  10 OR “m1=D”   10.0) AND (“m2=L”  5 OR “m2=D”  
5.0) AND (“m3=L” = 4 OR “m3=D” = 4.0)).
 
If filter value is in decimal format then we will have to make additional 
changes. If filter is something like JOB_ELAPSED_TIME  40.75 it will have to 
be converted to (“JOB_ELAPSED_TIME=L” = 41 OR “JOB_ELAPSED_TIME=D”  40.75). 
As you can see here, while matching a double value against column qualifier 
storing longs, I would need increase the value to closest integer and change 
filter to =. Likewise changes will be required for  (less than) and equal 
to(=) comparison as well.
 
However, I am not sure whether adding too many filters will cause any 
performance issue for HBase or not. Because with this solution, we will in 
essence be doubling the size of metric filters.
 
One thing we need to note though is that if we do adopt approach 2(including 
type in column qualifier), regex comparison might become an issue. Because 
theoretically regular expressions can become quite complex, so programmatically 
interpreting a regex and transforming it in a manner where it takes both long 
related column qualifier and double related column qualifier might induce bugs.
Maybe we can just support wildcard match(\*) or just do with prefix and 
substring filters.
 
Thoughts ?

However, we may want to match against only the latest version of the value for 
a metric.
In that case, the solution suggested above won’t work.

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena
 Attachments: YARN-4053-YARN-2928.01.patch


 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-24 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710633#comment-14710633
 ] 

Varun Saxena commented on YARN-4053:


Wanted to discuss so that we can reach a consensus on how to handle YARN-4053.

*Solution 1*: We can add a 1 byte flag as part of the metric value indicating 
whether we are storing integral value(0) or floating point value(1).
*Solution 2* : Another solution suggested is that type can be part of column 
qualifier say something like metric=l where l indicates long.

Another solution is to store everything as double. But would it be fair to 
impose this restriction on client while it reads data from ATS ? What if client 
is expecting a long and unable to handle a double.


The major issues surrounding different approaches are that what if client does 
not report metric values consistently(same metric data type). 

Now let us look at the scenarios where metric values come into picture.
*1.* While writing entity to HBase : Here, we need to consider that for the 
same entity, a particular metric can be reported in multiple write calls. 
So it is possible that in one write, all values for a particular metric are 
reported as long and in another write, all as floats. This can create 
inconsistency in both the solutions above (have different flags and encodings 
for same metric in Solution 1 and different column qualifiers for same metric 
in Solution 2).
We can add a valuetype field in TimelineMetric which indicates whether a set of 
values are long or float. And throw an exception in TimelineMetric at the time 
of adding value if types are not consistent. This will atleast ensure same data 
type for a particular write call.
But even here client should make sure that across writes they make sure data 
types are consistent. I think getting a row to find out column qualifier name 
or flags attached with the values wont be a viable option. 
So some sort of restriction on the part of the client(so that they send 
consistent data types for same metric) will have to be placed whether we adopt 
solution 1 or solution 2.
Is there some HBase API I am not aware of ?

*2.* While reading entity from HBase in the absence of any HBase filter : In 
this case there should be no issues in either solution 1 or solution 2. Because 
we read everything as bytes from HBase. We can do the appropriate conversion 
based on the flag or column qualifier name then.

*3.* While reading entity from HBase in the presence of HBase filters : We can 
have 2 kinds of HBase filters. One filter is to retrieve specific columns(to 
determine which metrics to return) and other one is to trim down the 
rows/entities to be returned based on metric value comparison.
The first class of filters which determine which columns to return, those 
should work in both the cases(Solution 1 and 2). 
Even in solution 2, because we use prefix filters as of now. If we use regex 
matching though, it might make things more complicated in case of Solution 2.

For the second set of filters, we would require to know data type of the metric 
value in both the proposed solutions. Because SingleColumnValueFilter requires 
exact column qualifier name(for Solution 2). And for solution 1 also we should 
know the data type of metric so that we can append the value to be compared 
against with the flag(so that BinaryComparator can be used).
If we add filters to our data object model, we can probably include data type 
in filters as well. But that again is dependent on client, whether it sends 
correct data type or not.


As we saw in point 1, we need to impose restriction on the client that it sends 
same data type for every metric. Frankly it should be easy for client as well. 
If for a metric, client expects float values, it will most likely use Double or 
Float.

Thoughts ? Or some other suggestions which can preclude the need for such a 
restriction. 

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena
 Attachments: YARN-4053-YARN-2928.01.patch


 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-20 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704368#comment-14704368
 ] 

Varun Saxena commented on YARN-4053:


bq. it might be good to restrict the numeric types the metric will support. 
long and double sounds good to me. Can add verification as you said.

bq. HBase already provides a facility to encode and decode between numbers and 
bytes
Yes I know. As I had to append one byte in front of the byte array, I moved the 
logic in Bytes.toBytes to a separate method. This was done to avoid creation of 
2 byte arrays(one inside Bytes.toBytes and one in ATS code) and henceforth 
copying over result from Bytes.toBytes to the byte array created inside ATS 
code. Although this is just 8 bytes. So maybe can do above.

bq. Also, instead of encoding the info whether this is an integral type vs. 
floating type into the value, it would be better to have this information in 
the column qualifier. 
I see some issue in having this info in column qualifier. Because certain HBase 
filters like SingleColumnValueFilter require exact column qualifier name. So we 
will have to again guess about the type(similar to current patch) when we use 
it. 
Probably we can discuss this offline and conclude there. Will send a mail.




 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena
 Attachments: YARN-4053-YARN-2928.01.patch


 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-19 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14703889#comment-14703889
 ] 

Sangjin Lee commented on YARN-4053:
---

I discussed this with [~jrottinghuis] a little more, and have some high level 
comments.

First off, HBase already provides a facility to encode and decode between 
numbers and bytes: {{Bytes.toBytes(long)}}, {{Bytes.toLong(byte[])}}, etc. We 
should use them instead of rolling our own.

Also, instead of encoding the info whether this is an integral type vs. 
floating type into the value, it would be better to have this information in 
the column qualifier. For example, if metric foo is of an integral type, then 
the column qualifier could contain ...=foo=l... (where l denotes a long 
value). That way, we can easily read out the type of values and deserialize 
appropriately. It would also remove any uncertainty around having different 
types of data for the same metric. There may be other variations of the idea.

Finally, I think it might be good to restrict the numeric types the metric will 
support. My proposal would be to limit it to {{Long}} and {{Double}}. They 
should be able to encompass pretty much all types of metrics. This would 
simplify support quite a bit. We can have verification in 
{{TimelineMetric.addValue()}} and so on to check the incoming type.

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena
 Attachments: YARN-4053-YARN-2928.01.patch


 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-17 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14700381#comment-14700381
 ] 

Sangjin Lee commented on YARN-4053:
---

And I do think that we need to support floating type values.

Another scenario to think about is what if users write metric values in an 
inconsistent manner. Suppose the user stored an integral value for a metric 
initially, but later attempted to store a floating value for the same metric. 
It sounds like it could be a silent failure? This should be a rare occurrence, 
but I think we need to give it some thought...

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena
 Attachments: YARN-4053-YARN-2928.01.patch


 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-17 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14700378#comment-14700378
 ] 

Sangjin Lee commented on YARN-4053:
---

Thanks [~varun_saxena] for pointing out an important issue. I would agree with 
[~gtCarrera9] that this is bit lower in priority compared to YARN-3814, but 
it's an important issue nonetheless.

I'm just curious (and perhaps this is a totally dumb question for a HBase 
newbie), is there a way to specify that the value type is a numeric type when 
we create the table or the column family? Does HBase itself support something 
like that?

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena
 Attachments: YARN-4053-YARN-2928.01.patch


 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-17 Thread Naganarasimha G R (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14700759#comment-14700759
 ] 

Naganarasimha G R commented on YARN-4053:
-

bq. place a restriction on client that it should send values in floating point 
format at all times if it wants to store some metric value as floating point. 
We can mention this in our documentation.
I think this approach is better as we will be able to have filters based on 
values too and less processing costs.

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena
 Attachments: YARN-4053-YARN-2928.01.patch


 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-17 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14700757#comment-14700757
 ] 

Varun Saxena commented on YARN-4053:


[~sjlee0],
bq. I'm just curious (and perhaps this is a totally dumb question for a HBase 
newbie), is there a way to specify that the value type is a numeric type when 
we create the table or the column family? Does HBase itself support something 
like that?
AFAIK, no there is no way to attach type with column qualifier or column 
family. HBase treats everything as just a sequence of bytes. It depends on the 
user how they encode or decode it.

bq. Another scenario to think about is what if users write metric values in an 
inconsistent manner. Suppose the user stored an integral value for a metric 
initially, but later attempted to store a floating value for the same metric. 
It sounds like it could be a silent failure? This should be a rare occurrence, 
but I think we need to give it some thought...
Yes I did consider this scenario. That is why I said we can place a restriction 
on client that it should send values in floating point format at all times if 
it wants to store some metric value as floating point. We can mention this in 
our documentation.

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena
 Attachments: YARN-4053-YARN-2928.01.patch


 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-17 Thread Li Lu (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699872#comment-14699872
 ] 

Li Lu commented on YARN-4053:
-

Hi [~varun_saxena], thanks for the patch! With regard to the POC, I thought we 
agreed on the general plan of the POC on a web UI connected to the timeline 
reader during our weekly standup discussion? If this is the case, I would 
personally give the RESTful API patch slightly higher priority since that is 
critical to the whole workflow of the reader/webUI interface? 

About this patch, I totally agree that we should directly store the byte 
representation of the numbers instead of using generic object mapper. Having 
looked at the patch, I have some general comments here. Maybe what we want here 
is a way to model the types of the timeline metrics, so that the type 
information can be carried over from TimelineMetric objects to the storage 
layer? We may have something like TimelineData.FLOAT, TimelineData.LONG, 
TimelineData.GENERIC_OBJECT, etc., so that we can easily transfer those 
messages? My main concern is on the ColumnPrefix descriptions, where we now use 
a boolean flag to indicate if the column is numeric or not. This will also help 
us to better organize the serialization and deserialization helper methods and 
all related tests. Let me know if this idea works here, thanks! 

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena
 Attachments: YARN-4053-YARN-2928.01.patch


 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-17 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699885#comment-14699885
 ] 

Varun Saxena commented on YARN-4053:


[~gtCarrera9], yes, will update a patch by tomorrow for YARN-3814 if no further 
comments come. Do you want it today ?

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena
 Attachments: YARN-4053-YARN-2928.01.patch


 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-17 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699907#comment-14699907
 ] 

Varun Saxena commented on YARN-4053:


Oh sorry in my original comment, I meant that we can carry this logic over to 
other columns(not only metrics). 
I agree we can include type information in TimelineMetric object itself. That 
will be better. By the way do you envisage metric values having anything other 
than float or long ? I think TimelineData.FLOAT and TimelineData.LONG should be 
enough. Thoughts ?

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena
 Attachments: YARN-4053-YARN-2928.01.patch


 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-17 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699883#comment-14699883
 ] 

Varun Saxena commented on YARN-4053:


bq. We may have something like TimelineData.FLOAT, TimelineData.LONG, 
TimelineData.GENERIC_OBJECT, etc., so that we can easily transfer those messages
Yes this can be done. I meant the same when I mentioned we can extend the logic 
in the patch attached.
Depends on what everyone agrees to.

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena
 Attachments: YARN-4053-YARN-2928.01.patch


 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-17 Thread Li Lu (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699888#comment-14699888
 ] 

Li Lu commented on YARN-4053:
-

Thanks! Tomorrow LGTM. 

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena
 Attachments: YARN-4053-YARN-2928.01.patch


 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-17 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699951#comment-14699951
 ] 

Varun Saxena commented on YARN-4053:


On second thoughts, all three types may make sense if we include filters as 
part of our object model and make client create and send them. Lets discuss 
this on Wednesday in weekly meeting.

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena
 Attachments: YARN-4053-YARN-2928.01.patch


 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-17 Thread Li Lu (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699971#comment-14699971
 ] 

Li Lu commented on YARN-4053:
-

bq. Oh sorry in my original comment, I meant that we can carry this logic over 
to other columns(not only metrics). 
This looks good. 

bq. I agree we can include type information in TimelineMetric object itself. 
That will be better. 
Actually I believe we *are* carrying type information in TimelineMetrics, in 
the original (boxed) Java form. For now I think we're fine to move with float 
and long. 

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena
 Attachments: YARN-4053-YARN-2928.01.patch


 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-16 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698602#comment-14698602
 ] 

Varun Saxena commented on YARN-4053:


This patch demonstrates the approach mentioned above and works for both 
integral and floating point values. But for floating point values, the 
restriction on part of the client is that it should send values in decimal 
format always otherwise when I add metric filters, matching will fail. I guess 
its a fair enough restriction to place.

In the patch, we can indicate that numerical values have to be stored per 
column/column prefix.
We can however extend this logic for all values and indicate if values to be 
stored are ASCII encoded as well, so that different kind of values can be 
stored differently in same column.

But there is no use case for this as of now, so haven't done so. 

I will remove the part about floating point numbers from patch, if we dont want 
it now.



 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena
 Attachments: YARN-4053-YARN-2928.01.patch


 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-16 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698608#comment-14698608
 ] 

Hadoop QA commented on YARN-4053:
-

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch |  16m  3s | Findbugs (version ) appears to 
be broken on YARN-2928. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to 
include 1 new or modified test files. |
| {color:green}+1{color} | javac |   7m 55s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |   9m 54s | There were no new javadoc 
warning messages. |
| {color:green}+1{color} | release audit |   0m 23s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle |   0m 16s | There were no new checkstyle 
issues. |
| {color:green}+1{color} | whitespace |   0m  1s | The patch has no lines that 
end in whitespace. |
| {color:green}+1{color} | install |   1m 26s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 40s | The patch built with 
eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   0m 52s | The patch does not introduce 
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests |   1m 22s | Tests passed in 
hadoop-yarn-server-timelineservice. |
| | |  38m 58s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12750693/YARN-4053-YARN-2928.01.patch
 |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-2928 / f40c735 |
| hadoop-yarn-server-timelineservice test log | 
https://builds.apache.org/job/PreCommit-YARN-Build/8853/artifact/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/8853/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/8853/console |


This message was automatically generated.

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena
 Attachments: YARN-4053-YARN-2928.01.patch


 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-15 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698209#comment-14698209
 ] 

Varun Saxena commented on YARN-4053:


Naga, yes double can also be used. 
But the problem with storing double is that we will be sending back a value 
with decimals and client may not be able to interpret it properly if its not 
expecting a decimal value(for integral values). IMO, we cant make assumptions 
about client because it may be assuming that a particular metric has been 
scored as an integral value in ATS. Although same problem will be with longs as 
well. But long is being thought of, as we are assuming we wont be supporting 
floating points as of now. A more concrete solution would be as under :

Assume solution 2 which I proposed above. Here we can have a flag (1 byte 
although 1 bit should be enough but for HBase storage 1 byte would be needed). 
This can indicate whether  a value is to be interpreted as long or double. We 
can take care of encoding/decoding while reading/writing from Hbase.

{noformat}
 18
-
   |   | |
   | 0 |(Integral value stored as long)  |
   |   | |
-

 18
-
   |   | |
   | 1 |  (Floating value stored as double)  |
   |   | |
-

If flag is 0, it indicates integral value and if 1 it indicates floating point 
value.

{noformat}

And we can place a restriction on clients saying that if they want a metric to 
be of floating point format, in all interactions with ATS the value should be 
in decimals i.e. 40 should be sent as 40.0 for instance. That will be a fair 
enough restriction to place.

However it all depends on whether we want to support floating points at the 
moment.




 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena

 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-14 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697449#comment-14697449
 ] 

Varun Saxena commented on YARN-4053:


Also for floating point metrics, query can be in integral form. This can create 
issues too. We should clearly document that query should also be in decimal 
representation for such metrics. That is, checking for condition like m1  40 
should be mean query from client should have filter as {{m1  40.0}} in REST 
API. So that its interpreted as a floating point number

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena

 Currently HBase implementation uses GenericObjectMapper is used to convert 
 and store values in backend HBase storage. This converts everything into a 
 string representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-14 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697556#comment-14697556
 ] 

Varun Saxena commented on YARN-4053:


bq. What kind of metrics do you have in mind that will have floating point 
numbers ?
There was some plan for reporting some cluster level metrics in future too, few 
of them would be floating point as well. Refer to json in YARN-3881
Also I remember some discussion during aggregation design regarding storing 
averages. Are we planning to calculate them on the fly instead ?
Moreover,TimelineMetric stores metric value as a {{java.lang.Number}}. This 
means we are saying metric can store a floating point value as well. As we have 
no control over systems outside YARN(say Tez), if they use ATS and publish a 
metric of floating type, I guess we should be able to handle it.

Thoughts ?

If it has been decided that metrics can only be integral values, then its fine. 
Wont have to take care of it then. Let me know.
Also, another key point we need to decide is that do we only support values 
till signed longs(8 bytes) ? 


 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena

 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-14 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697571#comment-14697571
 ] 

Varun Saxena commented on YARN-4053:


Tez may not be publishing any floating point metric as of now. I am not too 
sure about what all they publish. So probably there is no use case as of now. 
But if we do not support floating point numbers, then we should clearly 
document that we will only support integral values. And do the conversion in 
writer if any floating point value comes.

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena

 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-14 Thread Vrushali C (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697583#comment-14697583
 ] 

Vrushali C commented on YARN-4053:
--

Hmm good points. 

I think all metrics should be stored of the same type else we have to deal with 
knowing which metric is of which type and would need to store metadata to know 
how to read it back. Storing it as an ascii value is not good, we need to be 
able to query for things like less than greater than etc. 

My vote is for going with Longs for all metrics right now unless there is a 
very strong use case where only decimals will do. We truncate (cast down) 
decimals to long if we receive any, so 99.9 means 99. I realize this is 
restrictive but my thinking is that instead of trying to do everything for this 
current ATS release, let's go with Longs and see if we really need decimal 
precision. If we do, we can revisit and modify to accept more data types. cc 
[~jrottinghuis] 


 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena

 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-14 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697443#comment-14697443
 ] 

Varun Saxena commented on YARN-4053:


Storing metric values(which are numbers) as string is fine if we want to check 
them for equality.
But we have to support all relational operations for metrics. And that is where 
string representation doesnt work.
This is because in HBase, filters currently use lexicographical comparison. 
This means that with current mechanism to store metric values, a value of 4000 
will be judged as smaller than 60.


 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena

 Currently HBase implementation uses GenericObjectMapper is used to convert 
 and store values in backend HBase storage. This converts everything into a 
 string representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-14 Thread Vrushali C (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697500#comment-14697500
 ] 

Vrushali C commented on YARN-4053:
--

I think metric values should be stored (and read back) as Longs.

What kind of metrics do you have in mind that will have floating point numbers? 
Any percentages that we want to store? I don't think we really need that level 
of precision.

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena

 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-14 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697445#comment-14697445
 ] 

Varun Saxena commented on YARN-4053:


So to resolve this we need some other way of storing metric values. Options are 
as under :
# Keep the current way of storing metric values. And write a custom filter to 
match the values. But this would need the new filter to be deployed on all 
region servers. This solution hence may not be feasible. But if we do not want 
to do this, for lexicographic comparison to work, sizes of bytes compared 
should be equal.
# Store values as primitive types. That is, long as 8 bytes, integer as 4 bytes 
and so on. But this can create problems in lexicographical comparison too. Say 
metric m1 is stored as long. But a query to reader might be of the form {{m1  
4}}. As 4 will be interpreted as Integer, we will try to compare 4 
bytes against 8 bytes.
So the solution for this is to store every integral value as long(8 bytes) and 
floating point values as double. Same approach can be used while matching at 
reader side.
# But above solution may not work if we want to support BigInteger and 
BigDecimal values(i.e. numerical values  8 bytes). Although 8 bytes should be 
enough but aggregated values may exceed 8 bytes. In this case, we can probably 
decide values upto how many bytes do we need to support. 16 bytes, for that 
matter even 12 bytes should be more than enough for all realistic scenarios. 
While encoding we can do padding with zeroes in front if number is less than 16 
bytes.
# Another option can be to continue supporting string representation and 
restrict max number of digits we want to support before and after decimal 
point. Say 30 digits before decimal point and 3 after. We can pad rest of the 
bytes with zeroes while storing so that comparison can be done.

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena

 Currently HBase implementation uses GenericObjectMapper is used to convert 
 and store values in backend HBase storage. This converts everything into a 
 string representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-14 Thread Varun Saxena (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697454#comment-14697454
 ] 

Varun Saxena commented on YARN-4053:


cc [~sjlee0], [~djp], [~zjshen], [~vinodkv].
Thoughts ?
Will implement one of the options above depending on whatever is the consensus.

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena

 Currently HBase implementation uses GenericObjectMapper is used to convert 
 and store values in backend HBase storage. This converts everything into a 
 string representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-14 Thread Naganarasimha G R (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697967#comment-14697967
 ] 

Naganarasimha G R commented on YARN-4053:
-

[~vrushalic] how about double ?
I feel it would be the better as it too takes the same size of long (8 bytes) 
and supports decimals too ?

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena

 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-14 Thread Naganarasimha G R (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697968#comment-14697968
 ] 

Naganarasimha G R commented on YARN-4053:
-

[~vrushalic] how about double ?
I feel it would be the better as it too takes the same size of long (8 bytes) 
and supports decimals too ?

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena

 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage

2015-08-14 Thread Li Lu (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1469#comment-1469
 ] 

Li Lu commented on YARN-4053:
-

Hi [~varun_saxena], I agree this is a valid issue. Before we get deep 
involvement into this issue, I'm wondering if this is blocking any of our 
ongoing tasks to finish our planned POC of the reader and web UI? 

 Change the way metric values are stored in HBase Storage
 

 Key: YARN-4053
 URL: https://issues.apache.org/jira/browse/YARN-4053
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Varun Saxena
Assignee: Varun Saxena

 Currently HBase implementation uses GenericObjectMapper to convert and store 
 values in backend HBase storage. This converts everything into a string 
 representation(ASCII/UTF-8 encoded byte array).
 While this is fine in most cases, it does not quite serve our use case for 
 metrics. 
 So we need to decide how are we going to encode and decode metric values and 
 store them in HBase.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)