[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[ https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369809#comment-15369809 ] Hudson commented on YARN-3551: -- SUCCESS: Integrated in Hadoop-trunk-Commit #10074 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10074/]) YARN-3551. Consolidate data model change according to the backend (sjlee: rev fc8485d8934edf02bdef746d022d1125680e6efe) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timelineservice/TimelineEntity.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/records/timelineservice/TestTimelineServiceRecords.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TimelineServicePerformanceV2.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/TestTimelineServiceClientIntegration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timelineservice/TimelineMetric.java > Consolidate data model change according to the backend implementation > - > > Key: YARN-3551 > URL: https://issues.apache.org/jira/browse/YARN-3551 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: YARN-2928 > > Attachments: YARN-3551-YARN-2928.4.patch, > YARN-3551-YARN-2928.5.patch, YARN-3551.1.patch, YARN-3551.2.patch, > YARN-3551.3.patch > > > Based on the comments on > [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080] > and > [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098], > we need to change the data model to restrict the data type of > info/config/metric section. > 1. Info: the value could be all kinds object that is able to be > serialized/deserialized by jackson. > 2. Config: the value will always be assumed as String. > 3. Metric: single data or time series value have to be number for aggregation. > Other than that, info/start time/finish time of metric seem not to be > necessary for storage. They should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[
https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14527586#comment-14527586
]
Hadoop QA commented on YARN-3551:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12730280/YARN-3551-YARN-2928.5.patch
|
| Optional Tests | javac unit findbugs checkstyle javadoc |
| git revision | YARN-2928 / 557a395 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7693/console |
This message was automatically generated.
> Consolidate data model change according to the backend implementation
> -
>
> Key: YARN-3551
> URL: https://issues.apache.org/jira/browse/YARN-3551
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Fix For: YARN-2928
>
> Attachments: YARN-3551-YARN-2928.4.patch,
> YARN-3551-YARN-2928.5.patch, YARN-3551.1.patch, YARN-3551.2.patch,
> YARN-3551.3.patch
>
>
> Based on the comments on
> [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080]
> and
> [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098],
> we need to change the data model to restrict the data type of
> info/config/metric section.
> 1. Info: the value could be all kinds object that is able to be
> serialized/deserialized by jackson.
> 2. Config: the value will always be assumed as String.
> 3. Metric: single data or time series value have to be number for aggregation.
> Other than that, info/start time/finish time of metric seem not to be
> necessary for storage. They should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[ https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14527442#comment-14527442 ] Li Lu commented on YARN-3551: - The latest patch LGTM. +1 for commit. > Consolidate data model change according to the backend implementation > - > > Key: YARN-3551 > URL: https://issues.apache.org/jira/browse/YARN-3551 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-3551-YARN-2928.4.patch, > YARN-3551-YARN-2928.5.patch, YARN-3551.1.patch, YARN-3551.2.patch, > YARN-3551.3.patch > > > Based on the comments on > [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080] > and > [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098], > we need to change the data model to restrict the data type of > info/config/metric section. > 1. Info: the value could be all kinds object that is able to be > serialized/deserialized by jackson. > 2. Config: the value will always be assumed as String. > 3. Metric: single data or time series value have to be number for aggregation. > Other than that, info/start time/finish time of metric seem not to be > necessary for storage. They should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[ https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14527426#comment-14527426 ] Sangjin Lee commented on YARN-3551: --- LGTM. I've seen much flakiness with the test-patch.sh script, so those are probably not real errors. I'll wait for 1 hour for others to chime in (cc [~vrushalic], [~gtCarrera9]). If there are no further comments, I'll commit the patch. > Consolidate data model change according to the backend implementation > - > > Key: YARN-3551 > URL: https://issues.apache.org/jira/browse/YARN-3551 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-3551-YARN-2928.4.patch, > YARN-3551-YARN-2928.5.patch, YARN-3551.1.patch, YARN-3551.2.patch, > YARN-3551.3.patch > > > Based on the comments on > [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080] > and > [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098], > we need to change the data model to restrict the data type of > info/config/metric section. > 1. Info: the value could be all kinds object that is able to be > serialized/deserialized by jackson. > 2. Config: the value will always be assumed as String. > 3. Metric: single data or time series value have to be number for aggregation. > Other than that, info/start time/finish time of metric seem not to be > necessary for storage. They should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[ https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524144#comment-14524144 ] Li Lu commented on YARN-3551: - It seems like the 3 findbugs warnings were introduced in YARN-3039. I'm not sure if the unit tests are irrelevant. [~djp] could you please take a look at this, and maybe we can do a quick fix in either YARN-3039, or a separate JIRA? Thanks! > Consolidate data model change according to the backend implementation > - > > Key: YARN-3551 > URL: https://issues.apache.org/jira/browse/YARN-3551 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-3551-YARN-2928.4.patch, YARN-3551.1.patch, > YARN-3551.2.patch, YARN-3551.3.patch > > > Based on the comments on > [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080] > and > [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098], > we need to change the data model to restrict the data type of > info/config/metric section. > 1. Info: the value could be all kinds object that is able to be > serialized/deserialized by jackson. > 2. Config: the value will always be assumed as String. > 3. Metric: single data or time series value have to be number for aggregation. > Other than that, info/start time/finish time of metric seem not to be > necessary for storage. They should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[
https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14522960#comment-14522960
]
Hadoop QA commented on YARN-3551:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 16s | Pre-patch YARN-2928 compilation
is healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 47s | There were no new javac warning
messages. |
| {color:red}-1{color} | javadoc | 10m 7s | The applied patch generated 2
additional warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 32s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 9s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 4m 14s | The patch appears to introduce 3
new Findbugs (version 2.0.3) warnings. |
| {color:red}-1{color} | mapreduce tests | 62m 36s | Tests failed in
hadoop-mapreduce-client-jobclient. |
| {color:green}+1{color} | yarn tests | 0m 26s | Tests passed in
hadoop-yarn-api. |
| {color:red}-1{color} | yarn tests | 1m 39s | Tests failed in
hadoop-yarn-common. |
| {color:red}-1{color} | yarn tests | 0m 17s | Tests failed in
hadoop-yarn-server-tests. |
| | | 106m 42s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-common |
| | Inconsistent synchronization of
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.builder;
locked 92% of time Unsynchronized access at AllocateResponsePBImpl.java:92%
of time Unsynchronized access at AllocateResponsePBImpl.java:[line 391] |
| | Inconsistent synchronization of
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.proto;
locked 94% of time Unsynchronized access at AllocateResponsePBImpl.java:94%
of time Unsynchronized access at AllocateResponsePBImpl.java:[line 391] |
| | Inconsistent synchronization of
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.viaProto;
locked 94% of time Unsynchronized access at AllocateResponsePBImpl.java:94%
of time Unsynchronized access at AllocateResponsePBImpl.java:[line 391] |
| Failed unit tests | hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl |
| | hadoop.mapreduce.lib.jobcontrol.TestControlledJob |
| | hadoop.yarn.client.api.impl.TestTimelineClient |
| | hadoop.yarn.api.TestAllocateResponse |
| | hadoop.yarn.api.TestPBImplRecords |
| | hadoop.yarn.api.records.timelineservice.TestTimelineServiceRecords |
| | hadoop.yarn.logaggregation.TestAggregatedLogDeletionService |
| Failed build | hadoop-yarn-server-tests |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12729714/YARN-3551-YARN-2928.4.patch
|
| Optional Tests | javac unit findbugs checkstyle javadoc |
| git revision | YARN-2928 / b689f5d |
| javadoc |
https://builds.apache.org/job/PreCommit-YARN-Build/7568/artifact/patchprocess/diffJavadocWarnings.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/7568/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7568/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7568/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7568/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-tests test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7568/artifact/patchprocess/testrun_hadoop-yarn-server-tests.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/7568/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7568/console |
This message was automatically generated.
> Consolidate data model change according to the backend implementation
> -
>
> Key: YARN-3551
> URL: https://issues.apache.org/jira/browse/YARN-3551
>
[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[ https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14522785#comment-14522785 ] Sangjin Lee commented on YARN-3551: --- The latest patch looks good. Thanks for addressing the feedback and updating the patch [~zjshen]! There is one nit. I would prefer method and field names that don't include "timeSeries" in them, as the metric can be used for either a single value or time series. How about the following? timeSeries => values, getTimeSeriesJAXB() => getValuesJAXB(), getTimeSeries() => getValues(), setTimeSeries() => setValues(), addTimeSeries() => addValues(), addTimeSeriesData() => addValue() > Consolidate data model change according to the backend implementation > - > > Key: YARN-3551 > URL: https://issues.apache.org/jira/browse/YARN-3551 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-3551-YARN-2928.4.patch, YARN-3551.1.patch, > YARN-3551.2.patch, YARN-3551.3.patch > > > Based on the comments on > [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080] > and > [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098], > we need to change the data model to restrict the data type of > info/config/metric section. > 1. Info: the value could be all kinds object that is able to be > serialized/deserialized by jackson. > 2. Config: the value will always be assumed as String. > 3. Metric: single data or time series value have to be number for aggregation. > Other than that, info/start time/finish time of metric seem not to be > necessary for storage. They should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[ https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14522744#comment-14522744 ] Zhijie Shen commented on YARN-3551: --- Created a new patch to change timeline metric APIs according to the aforementioned comments. In addition, change the collection to TreeMap, and sort the data points according to timestamp in descending order. > Consolidate data model change according to the backend implementation > - > > Key: YARN-3551 > URL: https://issues.apache.org/jira/browse/YARN-3551 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-3551-YARN-2928.4.patch, YARN-3551.1.patch, > YARN-3551.2.patch, YARN-3551.3.patch > > > Based on the comments on > [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080] > and > [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098], > we need to change the data model to restrict the data type of > info/config/metric section. > 1. Info: the value could be all kinds object that is able to be > serialized/deserialized by jackson. > 2. Config: the value will always be assumed as String. > 3. Metric: single data or time series value have to be number for aggregation. > Other than that, info/start time/finish time of metric seem not to be > necessary for storage. They should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[ https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520484#comment-14520484 ] Zhijie Shen commented on YARN-3551: --- Cool, thanks for the comments, Sangjin. So the remaining unsettled issue is how we represent the single data point? Vrushali, did you have a chance to take a look at my last reply? > Consolidate data model change according to the backend implementation > - > > Key: YARN-3551 > URL: https://issues.apache.org/jira/browse/YARN-3551 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-3551.1.patch, YARN-3551.2.patch, YARN-3551.3.patch > > > Based on the comments on > [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080] > and > [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098], > we need to change the data model to restrict the data type of > info/config/metric section. > 1. Info: the value could be all kinds object that is able to be > serialized/deserialized by jackson. > 2. Config: the value will always be assumed as String. > 3. Metric: single data or time series value have to be number for aggregation. > Other than that, info/start time/finish time of metric seem not to be > necessary for storage. They should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[ https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520295#comment-14520295 ] Sangjin Lee commented on YARN-3551: --- I'm fine with going with using GenericOptionMapper for the serialization/deserialization of appropriate types. The generics is a suggestion for strengthening the types on the user side of things for the most part, so it may not be critical. > Consolidate data model change according to the backend implementation > - > > Key: YARN-3551 > URL: https://issues.apache.org/jira/browse/YARN-3551 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-3551.1.patch, YARN-3551.2.patch, YARN-3551.3.patch > > > Based on the comments on > [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080] > and > [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098], > we need to change the data model to restrict the data type of > info/config/metric section. > 1. Info: the value could be all kinds object that is able to be > serialized/deserialized by jackson. > 2. Config: the value will always be assumed as String. > 3. Metric: single data or time series value have to be number for aggregation. > Other than that, info/start time/finish time of metric seem not to be > necessary for storage. They should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[ https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518654#comment-14518654 ] Zhijie Shen commented on YARN-3551: --- bq. Also, think along with singleData, we also need a timestamp member variable, else we would not know which timestamp this singleData value belongs to. If we want to know the timestamp of the value, what's difference between single data and time series? Or can I rephrase your proposal in another way: there exists two modes of a metric: 1. We remember all history values of one metric. 2. We only remember the last value of one metric. So at the client for the first one, we use the data model api to buffer multiple data points of time series, while for the second one, we keep update the single point with the latest value. Once it is handed over to the server, for the first one, the server will persisted all the points in HBase by using version, while for the second one, the server will only track the latest version of the point. bq. How about making the timeline metric generic? I'm +0 about using generic type: \+ It's good to restrict one metric just contains one type of value. \- The caller will only know the object is TimelineMetric but won't know what the "?" is before checking the object itself. It's no easier than inspecting the data directly. \- It may not be complete consistent with the way jackson marshal and unmarshal the JSON object. For example, if we send a TimelineMetric(1L, 2L, 3L, 4L), we will actually get TimelineMetric(1, 2, 3, 4) according jasckon's rule of ser/des. bq. Hmm, yes, we do need a datatype per metric to be stored, this will have to be considered in the writer impls. Please take a look at my comment on [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14517838&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14517838]. Once we receive the data at the server side, we know what kind of number the metric is. To ser/des with the backend, we can make use of GenericObjectMapper. > Consolidate data model change according to the backend implementation > - > > Key: YARN-3551 > URL: https://issues.apache.org/jira/browse/YARN-3551 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-3551.1.patch, YARN-3551.2.patch, YARN-3551.3.patch > > > Based on the comments on > [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080] > and > [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098], > we need to change the data model to restrict the data type of > info/config/metric section. > 1. Info: the value could be all kinds object that is able to be > serialized/deserialized by jackson. > 2. Config: the value will always be assumed as String. > 3. Metric: single data or time series value have to be number for aggregation. > Other than that, info/start time/finish time of metric seem not to be > necessary for storage. They should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[ https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518583#comment-14518583 ] Vrushali C commented on YARN-3551: -- + 1 to the constructor and templated code above. bq. In the storage layer, perhaps one could store the type look-up upon seeing the metric for the first time? This is going to make it a bit complex isn't it. We wouldn't know if we are seeing the metric for the first time unless we look it up each time. Or unless some boolean is passed in while calling the TimelineWriter api. Which again makes me lean towards having a completely new API for writing metrics solely, in addition to the write for TimelineEntity. > Consolidate data model change according to the backend implementation > - > > Key: YARN-3551 > URL: https://issues.apache.org/jira/browse/YARN-3551 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-3551.1.patch, YARN-3551.2.patch, YARN-3551.3.patch > > > Based on the comments on > [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080] > and > [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098], > we need to change the data model to restrict the data type of > info/config/metric section. > 1. Info: the value could be all kinds object that is able to be > serialized/deserialized by jackson. > 2. Config: the value will always be assumed as String. > 3. Metric: single data or time series value have to be number for aggregation. > Other than that, info/start time/finish time of metric seem not to be > necessary for storage. They should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[ https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518575#comment-14518575 ] Vrushali C commented on YARN-3551: -- Hmm, yes, we do need a datatype per metric to be stored, this will have to be considered in the writer impls. Perhaps for the testing part, we could write longs for now. . Also, in the unit test in [~zjshen]'s latest patch, for the same metric, once a long is written, then a double is written at a different timestamp. This is probably misleading. > Consolidate data model change according to the backend implementation > - > > Key: YARN-3551 > URL: https://issues.apache.org/jira/browse/YARN-3551 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-3551.1.patch, YARN-3551.2.patch, YARN-3551.3.patch > > > Based on the comments on > [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080] > and > [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098], > we need to change the data model to restrict the data type of > info/config/metric section. > 1. Info: the value could be all kinds object that is able to be > serialized/deserialized by jackson. > 2. Config: the value will always be assumed as String. > 3. Metric: single data or time series value have to be number for aggregation. > Other than that, info/start time/finish time of metric seem not to be > necessary for storage. They should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[ https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518569#comment-14518569 ] Vrushali C commented on YARN-3551: -- Thanks [~zjshen] for the jira and the patch. I was wondering about the TimelineMetric class. Do you think it might sound better if we call it latestData or mostRecentData instead of singleData? Also, think along with singleData, we also need a timestamp member variable, else we would not know which timestamp this singleData value belongs to. Another way of doing this would be to keep adding to the time series member of TimelineMetric and modify the write API to write only the latest metric value. I am wondering if this may actually be better than creating a timeline entity object with a single metric value each time. Metrics are updated very frequently and for long running apps, we will end up with a lot of object creation churn. > Consolidate data model change according to the backend implementation > - > > Key: YARN-3551 > URL: https://issues.apache.org/jira/browse/YARN-3551 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-3551.1.patch, YARN-3551.2.patch, YARN-3551.3.patch > > > Based on the comments on > [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080] > and > [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098], > we need to change the data model to restrict the data type of > info/config/metric section. > 1. Info: the value could be all kinds object that is able to be > serialized/deserialized by jackson. > 2. Config: the value will always be assumed as String. > 3. Metric: single data or time series value have to be number for aggregation. > Other than that, info/start time/finish time of metric seem not to be > necessary for storage. They should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[ https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518494#comment-14518494 ] Sangjin Lee commented on YARN-3551: --- I don't think those two things are mutually exclusive. In the storage layer, perhaps one could store the type look-up upon seeing the metric for the first time? If we go this route, it would be nice to retain the above form of the constructor, because it would be impossible to specify incorrect type information. > Consolidate data model change according to the backend implementation > - > > Key: YARN-3551 > URL: https://issues.apache.org/jira/browse/YARN-3551 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-3551.1.patch, YARN-3551.2.patch, YARN-3551.3.patch > > > Based on the comments on > [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080] > and > [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098], > we need to change the data model to restrict the data type of > info/config/metric section. > 1. Info: the value could be all kinds object that is able to be > serialized/deserialized by jackson. > 2. Config: the value will always be assumed as String. > 3. Metric: single data or time series value have to be number for aggregation. > Other than that, info/start time/finish time of metric seem not to be > necessary for storage. They should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[ https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518450#comment-14518450 ] Li Lu commented on YARN-3551: - OK, I didn't say it clear enough but this is the solution I was talking about. In order to trace the type of a metric, we need to require type information explicitly. We also need to store this information. Storing types for each metric may cause significant redundancy. To improve space efficiency we may need to set up a per-metric table to store their type information? > Consolidate data model change according to the backend implementation > - > > Key: YARN-3551 > URL: https://issues.apache.org/jira/browse/YARN-3551 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-3551.1.patch, YARN-3551.2.patch, YARN-3551.3.patch > > > Based on the comments on > [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080] > and > [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098], > we need to change the data model to restrict the data type of > info/config/metric section. > 1. Info: the value could be all kinds object that is able to be > serialized/deserialized by jackson. > 2. Config: the value will always be assumed as String. > 3. Metric: single data or time series value have to be number for aggregation. > Other than that, info/start time/finish time of metric seem not to be > necessary for storage. They should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[
https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518414#comment-14518414
]
Sangjin Lee commented on YARN-3551:
---
I was going to follow up. The generics alone is not going to give you type
information at runtime, as generics are erased at runtime as you know. The
above would give you type safety mostly from the user point of view.
I saw Vrushali's comment about serializing/deserializing appropriate type in
another JIRA. We could handle the type by modifying the above slightly:
{code}
public class TimelineMetric {
...
private final Class type;
private T singleData;
private HashMap timeSeries;
public TimelineMetric(Class type) {
this.type = type;
}
...
}
{code}
> Consolidate data model change according to the backend implementation
> -
>
> Key: YARN-3551
> URL: https://issues.apache.org/jira/browse/YARN-3551
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Attachments: YARN-3551.1.patch, YARN-3551.2.patch, YARN-3551.3.patch
>
>
> Based on the comments on
> [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080]
> and
> [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098],
> we need to change the data model to restrict the data type of
> info/config/metric section.
> 1. Info: the value could be all kinds object that is able to be
> serialized/deserialized by jackson.
> 2. Config: the value will always be assumed as String.
> 3. Metric: single data or time series value have to be number for aggregation.
> Other than that, info/start time/finish time of metric seem not to be
> necessary for storage. They should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[ https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518360#comment-14518360 ] Li Lu commented on YARN-3551: - Hi [~sjlee0]! I agree it would be nice to have more strict type enforcement with templates, but right now I'm not very clear about how this would impact the writer/reader implementation. Upon storage, how can we choose the right serialization method for the type? Meanwhile, to reconstruct the data in readers, maybe we need to store metadata about the type information for each metric object? > Consolidate data model change according to the backend implementation > - > > Key: YARN-3551 > URL: https://issues.apache.org/jira/browse/YARN-3551 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-3551.1.patch, YARN-3551.2.patch, YARN-3551.3.patch > > > Based on the comments on > [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080] > and > [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098], > we need to change the data model to restrict the data type of > info/config/metric section. > 1. Info: the value could be all kinds object that is able to be > serialized/deserialized by jackson. > 2. Config: the value will always be assumed as String. > 3. Metric: single data or time series value have to be number for aggregation. > Other than that, info/start time/finish time of metric seem not to be > necessary for storage. They should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[
https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518338#comment-14518338
]
Sangjin Lee commented on YARN-3551:
---
Thanks for the patch [~zjshen]!
How about making the timeline metric generic? I'm not 100% certain that will
play nice with serialization/deserialization, but it would be great to be able
to specify which specific numeric type a given metric is and enforce it
strictly:
{code}
public class TimelineMetric {
...
private T singleData;
private HashMap timeSeries;
...
}
{code}
Thoughts?
> Consolidate data model change according to the backend implementation
> -
>
> Key: YARN-3551
> URL: https://issues.apache.org/jira/browse/YARN-3551
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Attachments: YARN-3551.1.patch, YARN-3551.2.patch, YARN-3551.3.patch
>
>
> Based on the comments on
> [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080]
> and
> [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098],
> we need to change the data model to restrict the data type of
> info/config/metric section.
> 1. Info: the value could be all kinds object that is able to be
> serialized/deserialized by jackson.
> 2. Config: the value will always be assumed as String.
> 3. Metric: single data or time series value have to be number for aggregation.
> Other than that, info/start time/finish time of metric seem not to be
> necessary for storage. They should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[ https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516422#comment-14516422 ] Li Lu commented on YARN-3551: - Hi [~zjshen], thanks for the update. The latest patch LGTM. About startTime and endTime, I'll modify my implementation according to this. Since this is a data model related change, let's keep it hanging there for a little while, and then commit it? > Consolidate data model change according to the backend implementation > - > > Key: YARN-3551 > URL: https://issues.apache.org/jira/browse/YARN-3551 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-3551.1.patch, YARN-3551.2.patch > > > Based on the comments on > [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080] > and > [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098], > we need to change the data model to restrict the data type of > info/config/metric section. > 1. Info: the value could be all kinds object that is able to be > serialized/deserialized by jackson. > 2. Config: the value will always be assumed as String. > 3. Metric: single data or time series value have to be number for aggregation. > Other than that, info/start time/finish time of metric seem not to be > necessary for storage. They should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[ https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516359#comment-14516359 ] Zhijie Shen commented on YARN-3551: --- Li, thanks for the comments. See the following response. bq. Since singleData is a Number now, what is the consideration to return it as an Object in getSingleData? It's a bug, and fixed in the new patch. bq. Since a metric object can only store one type of data (timeseries or single data), maybe it's helpful to have a method to distinguish these two cases? There still exists the case that both single data and timeseries are left empty. So can we just go ahead to check if the data exist or not? bq. checkIfTheOhterExists appears to be a typo. Fixed. bq. We're removing startTime and endTime here, which are used as primary keys for Phoenix based storage. After this JIRA we need to find an alternative way to store metrics in Phoenix. I'm marking this JIRA as blocking YARN-3134. Starttime and endtime is used to define a period to query the aggregated information during it (see comments in [Comment 1 in YARN-3041|https://issues.apache.org/jira/browse/YARN-3041?focusedCommentId=14325492&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14325492] and [Comment 2 in YARN-3041|https://issues.apache.org/jira/browse/YARN-3041?focusedCommentId=14327011&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14327011]. Hence I don't think they should be used as part of the primary key. I propose to remove it because I think starttime and endtime should be given as the request's params. The response could just be the metric object containing the statistic information during this given period. > Consolidate data model change according to the backend implementation > - > > Key: YARN-3551 > URL: https://issues.apache.org/jira/browse/YARN-3551 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-3551.1.patch, YARN-3551.2.patch > > > Based on the comments on > [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080] > and > [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098], > we need to change the data model to restrict the data type of > info/config/metric section. > 1. Info: the value could be all kinds object that is able to be > serialized/deserialized by jackson. > 2. Config: the value will always be assumed as String. > 3. Metric: single data or time series value have to be number for aggregation. > Other than that, info/start time/finish time of metric seem not to be > necessary for storage. They should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3551) Consolidate data model change according to the backend implementation
[
https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516073#comment-14516073
]
Li Lu commented on YARN-3551:
-
Hi [~zjshen], thanks for the patch! Some quick comments and thoughts:
# Since singleData is a Number now, what is the consideration to return it as
an Object in getSingleData?
# Since a metric object can only store one type of data (timeseries or single
data), maybe it's helpful to have a method to distinguish these two cases?
# {{checkIfTheOhterExists}} appears to be a typo.
# We're removing startTime and endTime here, which are used as primary keys for
Phoenix based storage. After this JIRA we need to find an alternative way to
store metrics in Phoenix. I'm marking this JIRA as blocking YARN-3134.
> Consolidate data model change according to the backend implementation
> -
>
> Key: YARN-3551
> URL: https://issues.apache.org/jira/browse/YARN-3551
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Attachments: YARN-3551.1.patch
>
>
> Based on the comments on
> [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080]
> and
> [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098],
> we need to change the data model to restrict the data type of
> info/config/metric section.
> 1. Info: the value could be all kinds object that is able to be
> serialized/deserialized by jackson.
> 2. Config: the value will always be assumed as String.
> 3. Metric: single data or time series value have to be number for aggregation.
> Other than that, info/start time/finish time of metric seem not to be
> necessary for storage. They should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
