[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[ https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369787#comment-15369787 ] Hudson commented on YARN-4178: -- SUCCESS: Integrated in Hadoop-trunk-Commit #10074 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10074/]) YARN-4178. [storage implementation] app id as string in row keys can (sjlee: rev b51d0fef56a59b15489f5b932025718b4e9613d2) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/ApplicationEntityReader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/TimelineWriterUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestHBaseTimelineStorage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowActivityRowKey.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/entity/EntityRowKey.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/TestHBaseStorageFlowActivity.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunRowKey.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseTimelineWriterImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowActivityColumnPrefix.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/application/ApplicationRowKey.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/Separator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowScanner.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/TestTimelineWriterUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/FileSystemTimelineReaderImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/TestTimelineStorageUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/GenericEntityReader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunColumn.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunCoprocessor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/apptoflow/AppToFlowRowKey.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/TimelineReaderUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/TimelineStorageUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunColumnPrefix.java > [storage implementation] app id as string in row keys can cause incorrect > ordering > -- > > Key: YARN-4178 > URL: https://issues.apache.org/jira/browse/YARN-4178 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[ https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946465#comment-14946465 ] Varun Saxena commented on YARN-4178: Thanks for the commit [~sjlee0] and others for the reviews > [storage implementation] app id as string in row keys can cause incorrect > ordering > -- > > Key: YARN-4178 > URL: https://issues.apache.org/jira/browse/YARN-4178 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Fix For: YARN-2928 > > Attachments: YARN-4178-YARN-2928.01.patch, > YARN-4178-YARN-2928.02.patch, YARN-4178-YARN-2928.03.patch, > YARN-4178-YARN-2928.04.patch, YARN-4178-YARN-2928.05.patch > > > Currently the app id is used in various places as part of row keys. However, > currently they are treated as strings. This will cause a problem with > ordering when the id portion of the app id rolls over to the next digit. > For example, "app_1234567890_1" will be considered *earlier* than > "app_1234567890_". We should correct this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[ https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945480#comment-14945480 ] Varun Saxena commented on YARN-4178: Ok. But then should we rename the other one as invertLong ? > [storage implementation] app id as string in row keys can cause incorrect > ordering > -- > > Key: YARN-4178 > URL: https://issues.apache.org/jira/browse/YARN-4178 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4178-YARN-2928.01.patch, > YARN-4178-YARN-2928.02.patch, YARN-4178-YARN-2928.03.patch, > YARN-4178-YARN-2928.04.patch > > > Currently the app id is used in various places as part of row keys. However, > currently they are treated as strings. This will cause a problem with > ordering when the id portion of the app id rolls over to the next digit. > For example, "app_1234567890_1" will be considered *earlier* than > "app_1234567890_". We should correct this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[
https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945475#comment-14945475
]
Sangjin Lee commented on YARN-4178:
---
The latest patch looks good to me for the most part. I just had one minor point.
(TimelineStorageUtils.java)
- l.169: I might prefer to name this method a little differently. Currently,
the right method will be picked depending on the type of the input argument.
However, I see possibility of making a mistake and accidentally getting an
incorrect result. How about differentiating this strongly, e.g. {{invertInt()}}?
> [storage implementation] app id as string in row keys can cause incorrect
> ordering
> --
>
> Key: YARN-4178
> URL: https://issues.apache.org/jira/browse/YARN-4178
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
> Attachments: YARN-4178-YARN-2928.01.patch,
> YARN-4178-YARN-2928.02.patch, YARN-4178-YARN-2928.03.patch,
> YARN-4178-YARN-2928.04.patch
>
>
> Currently the app id is used in various places as part of row keys. However,
> currently they are treated as strings. This will cause a problem with
> ordering when the id portion of the app id rolls over to the next digit.
> For example, "app_1234567890_1" will be considered *earlier* than
> "app_1234567890_". We should correct this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[ https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945485#comment-14945485 ] Sangjin Lee commented on YARN-4178: --- That's fine too. I'm comfortable either way (invert/invertInt vs. invertLong/invertInt). > [storage implementation] app id as string in row keys can cause incorrect > ordering > -- > > Key: YARN-4178 > URL: https://issues.apache.org/jira/browse/YARN-4178 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4178-YARN-2928.01.patch, > YARN-4178-YARN-2928.02.patch, YARN-4178-YARN-2928.03.patch, > YARN-4178-YARN-2928.04.patch > > > Currently the app id is used in various places as part of row keys. However, > currently they are treated as strings. This will cause a problem with > ordering when the id portion of the app id rolls over to the next digit. > For example, "app_1234567890_1" will be considered *earlier* than > "app_1234567890_". We should correct this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[ https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945776#comment-14945776 ] Vrushali C commented on YARN-4178: -- Patch v5 looks good to me too. > [storage implementation] app id as string in row keys can cause incorrect > ordering > -- > > Key: YARN-4178 > URL: https://issues.apache.org/jira/browse/YARN-4178 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4178-YARN-2928.01.patch, > YARN-4178-YARN-2928.02.patch, YARN-4178-YARN-2928.03.patch, > YARN-4178-YARN-2928.04.patch, YARN-4178-YARN-2928.05.patch > > > Currently the app id is used in various places as part of row keys. However, > currently they are treated as strings. This will cause a problem with > ordering when the id portion of the app id rolls over to the next digit. > For example, "app_1234567890_1" will be considered *earlier* than > "app_1234567890_". We should correct this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[
https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945643#comment-14945643
]
Hadoop QA commented on YARN-4178:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 38s | Findbugs (version ) appears to
be broken on YARN-2928. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 4 new or modified test files. |
| {color:green}+1{color} | javac | 8m 16s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 18s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 17s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 5s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 41s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 55s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 3m 32s | Tests passed in
hadoop-yarn-server-timelineservice. |
| | | 42m 47s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12765222/YARN-4178-YARN-2928.05.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-2928 / 09c3576 |
| hadoop-yarn-server-timelineservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9363/artifact/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9363/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9363/console |
This message was automatically generated.
> [storage implementation] app id as string in row keys can cause incorrect
> ordering
> --
>
> Key: YARN-4178
> URL: https://issues.apache.org/jira/browse/YARN-4178
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
> Attachments: YARN-4178-YARN-2928.01.patch,
> YARN-4178-YARN-2928.02.patch, YARN-4178-YARN-2928.03.patch,
> YARN-4178-YARN-2928.04.patch, YARN-4178-YARN-2928.05.patch
>
>
> Currently the app id is used in various places as part of row keys. However,
> currently they are treated as strings. This will cause a problem with
> ordering when the id portion of the app id rolls over to the next digit.
> For example, "app_1234567890_1" will be considered *earlier* than
> "app_1234567890_". We should correct this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[ https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945750#comment-14945750 ] Sangjin Lee commented on YARN-4178: --- The latest patch LGTM. I'll wait a little while before I commit to give other folks to chime in. > [storage implementation] app id as string in row keys can cause incorrect > ordering > -- > > Key: YARN-4178 > URL: https://issues.apache.org/jira/browse/YARN-4178 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4178-YARN-2928.01.patch, > YARN-4178-YARN-2928.02.patch, YARN-4178-YARN-2928.03.patch, > YARN-4178-YARN-2928.04.patch, YARN-4178-YARN-2928.05.patch > > > Currently the app id is used in various places as part of row keys. However, > currently they are treated as strings. This will cause a problem with > ordering when the id portion of the app id rolls over to the next digit. > For example, "app_1234567890_1" will be considered *earlier* than > "app_1234567890_". We should correct this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[ https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943703#comment-14943703 ] Sangjin Lee commented on YARN-4178: --- +1 with consolidating WriterUtils and ReaderUtils. > [storage implementation] app id as string in row keys can cause incorrect > ordering > -- > > Key: YARN-4178 > URL: https://issues.apache.org/jira/browse/YARN-4178 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4178-YARN-2928.01.patch, > YARN-4178-YARN-2928.02.patch > > > Currently the app id is used in various places as part of row keys. However, > currently they are treated as strings. This will cause a problem with > ordering when the id portion of the app id rolls over to the next digit. > For example, "app_1234567890_1" will be considered *earlier* than > "app_1234567890_". We should correct this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[ https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943778#comment-14943778 ] Vrushali C commented on YARN-4178: -- bq. I think we can have a single class TimelineStorageUtils and remove TimelineWriterUtils and TimelineReaderUtils. Sounds good. I guess it will be a big class but shouldn't be a problem. > [storage implementation] app id as string in row keys can cause incorrect > ordering > -- > > Key: YARN-4178 > URL: https://issues.apache.org/jira/browse/YARN-4178 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4178-YARN-2928.01.patch, > YARN-4178-YARN-2928.02.patch > > > Currently the app id is used in various places as part of row keys. However, > currently they are treated as strings. This will cause a problem with > ordering when the id portion of the app id rolls over to the next digit. > For example, "app_1234567890_1" will be considered *earlier* than > "app_1234567890_". We should correct this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[ https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943863#comment-14943863 ] Varun Saxena commented on YARN-4178: Depending on order in which patches go in, YARN-4178 or YARN-3864 will require rebase > [storage implementation] app id as string in row keys can cause incorrect > ordering > -- > > Key: YARN-4178 > URL: https://issues.apache.org/jira/browse/YARN-4178 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4178-YARN-2928.01.patch, > YARN-4178-YARN-2928.02.patch, YARN-4178-YARN-2928.03.patch > > > Currently the app id is used in various places as part of row keys. However, > currently they are treated as strings. This will cause a problem with > ordering when the id portion of the app id rolls over to the next digit. > For example, "app_1234567890_1" will be considered *earlier* than > "app_1234567890_". We should correct this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[ https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943957#comment-14943957 ] Sangjin Lee commented on YARN-4178: --- I just committed YARN-3864. Could you please rebase this patch? Thanks. > [storage implementation] app id as string in row keys can cause incorrect > ordering > -- > > Key: YARN-4178 > URL: https://issues.apache.org/jira/browse/YARN-4178 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4178-YARN-2928.01.patch, > YARN-4178-YARN-2928.02.patch, YARN-4178-YARN-2928.03.patch > > > Currently the app id is used in various places as part of row keys. However, > currently they are treated as strings. This will cause a problem with > ordering when the id portion of the app id rolls over to the next digit. > For example, "app_1234567890_1" will be considered *earlier* than > "app_1234567890_". We should correct this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[
https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943973#comment-14943973
]
Hadoop QA commented on YARN-4178:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 18m 29s | Findbugs (version ) appears to
be broken on YARN-2928. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 4 new or modified test files. |
| {color:green}+1{color} | javac | 9m 14s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 11m 40s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 26s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 18s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 5s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 44s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 45s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 2s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 2m 53s | Tests passed in
hadoop-yarn-server-timelineservice. |
| | | 46m 44s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12765047/YARN-4178-YARN-2928.03.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-2928 / a95b8f5 |
| hadoop-yarn-server-timelineservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9352/artifact/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9352/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9352/console |
This message was automatically generated.
> [storage implementation] app id as string in row keys can cause incorrect
> ordering
> --
>
> Key: YARN-4178
> URL: https://issues.apache.org/jira/browse/YARN-4178
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
> Attachments: YARN-4178-YARN-2928.01.patch,
> YARN-4178-YARN-2928.02.patch, YARN-4178-YARN-2928.03.patch
>
>
> Currently the app id is used in various places as part of row keys. However,
> currently they are treated as strings. This will cause a problem with
> ordering when the id portion of the app id rolls over to the next digit.
> For example, "app_1234567890_1" will be considered *earlier* than
> "app_1234567890_". We should correct this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[ https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14942973#comment-14942973 ] Varun Saxena commented on YARN-4178: bq. To that effect, if you’d like, we can rename TimelineWriterUtils to TimelineStorageUtils so that both reader and writer can use functions from this. Also,let’s have the invert(long) and invert(int) functions in the same util class, instead of adding in a new util class. I think we can have a single class TimelineStorageUtils and remove TimelineWriterUtils and TimelineReaderUtils. > [storage implementation] app id as string in row keys can cause incorrect > ordering > -- > > Key: YARN-4178 > URL: https://issues.apache.org/jira/browse/YARN-4178 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4178-YARN-2928.01.patch, > YARN-4178-YARN-2928.02.patch > > > Currently the app id is used in various places as part of row keys. However, > currently they are treated as strings. This will cause a problem with > ordering when the id portion of the app id rolls over to the next digit. > For example, "app_1234567890_1" will be considered *earlier* than > "app_1234567890_". We should correct this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[ https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14942972#comment-14942972 ] Varun Saxena commented on YARN-4178: bq. Why do we need any util classes for this, can't an AppId class handle this by itself (convert from string to byte representation and back)? Should we be adding code specific to ATS and that too HBase implementation of ATS into a class(ApplicationId) which is used all across YARN ? > [storage implementation] app id as string in row keys can cause incorrect > ordering > -- > > Key: YARN-4178 > URL: https://issues.apache.org/jira/browse/YARN-4178 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4178-YARN-2928.01.patch, > YARN-4178-YARN-2928.02.patch > > > Currently the app id is used in various places as part of row keys. However, > currently they are treated as strings. This will cause a problem with > ordering when the id portion of the app id rolls over to the next digit. > For example, "app_1234567890_1" will be considered *earlier* than > "app_1234567890_". We should correct this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[ https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944130#comment-14944130 ] Varun Saxena commented on YARN-4178: New patch updated. > [storage implementation] app id as string in row keys can cause incorrect > ordering > -- > > Key: YARN-4178 > URL: https://issues.apache.org/jira/browse/YARN-4178 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4178-YARN-2928.01.patch, > YARN-4178-YARN-2928.02.patch, YARN-4178-YARN-2928.03.patch, > YARN-4178-YARN-2928.04.patch > > > Currently the app id is used in various places as part of row keys. However, > currently they are treated as strings. This will cause a problem with > ordering when the id portion of the app id rolls over to the next digit. > For example, "app_1234567890_1" will be considered *earlier* than > "app_1234567890_". We should correct this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[
https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944123#comment-14944123
]
Hadoop QA commented on YARN-4178:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 18m 12s | Findbugs (version ) appears to
be broken on YARN-2928. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 4 new or modified test files. |
| {color:red}-1{color} | javac | 4m 31s | The patch appears to cause the
build to fail. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12765067/YARN-4178-YARN-2928.04.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-2928 / 09c3576 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9353/console |
This message was automatically generated.
> [storage implementation] app id as string in row keys can cause incorrect
> ordering
> --
>
> Key: YARN-4178
> URL: https://issues.apache.org/jira/browse/YARN-4178
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
> Attachments: YARN-4178-YARN-2928.01.patch,
> YARN-4178-YARN-2928.02.patch, YARN-4178-YARN-2928.03.patch,
> YARN-4178-YARN-2928.04.patch
>
>
> Currently the app id is used in various places as part of row keys. However,
> currently they are treated as strings. This will cause a problem with
> ordering when the id portion of the app id rolls over to the next digit.
> For example, "app_1234567890_1" will be considered *earlier* than
> "app_1234567890_". We should correct this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[
https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944190#comment-14944190
]
Hadoop QA commented on YARN-4178:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 2s | Findbugs (version ) appears to
be broken on YARN-2928. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 4 new or modified test files. |
| {color:green}+1{color} | javac | 8m 8s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 22s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 17s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 4s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 40s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 56s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 2m 49s | Tests passed in
hadoop-yarn-server-timelineservice. |
| | | 41m 24s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12765073/YARN-4178-YARN-2928.04.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-2928 / 09c3576 |
| hadoop-yarn-server-timelineservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9354/artifact/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9354/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9354/console |
This message was automatically generated.
> [storage implementation] app id as string in row keys can cause incorrect
> ordering
> --
>
> Key: YARN-4178
> URL: https://issues.apache.org/jira/browse/YARN-4178
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
> Attachments: YARN-4178-YARN-2928.01.patch,
> YARN-4178-YARN-2928.02.patch, YARN-4178-YARN-2928.03.patch,
> YARN-4178-YARN-2928.04.patch
>
>
> Currently the app id is used in various places as part of row keys. However,
> currently they are treated as strings. This will cause a problem with
> ordering when the id portion of the app id rolls over to the next digit.
> For example, "app_1234567890_1" will be considered *earlier* than
> "app_1234567890_". We should correct this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[ https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938953#comment-14938953 ] Vrushali C commented on YARN-4178: -- Thanks [~varun_saxena] for the patch. Overall, LGTM. A couple of observations: It would be good to have encodeAppId and decodeAppId in the same class instead of two different classes. To that effect, if you’d like, we can rename TimelineWriterUtils to TimelineStorageUtils so that both reader and writer can use functions from this. Also,let’s have the invert(long) and invert(int) functions in the same util class, instead of adding in a new util class. While I do think we should store the “application_” prefix (if/when yarn starts allowing configurable prefixes so that we can see something like "spark__" or "tez__" on the cluster etc), I don’t want to hold up the jira for that since we could add it in later as we see fit. > [storage implementation] app id as string in row keys can cause incorrect > ordering > -- > > Key: YARN-4178 > URL: https://issues.apache.org/jira/browse/YARN-4178 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4178-YARN-2928.01.patch, > YARN-4178-YARN-2928.02.patch > > > Currently the app id is used in various places as part of row keys. However, > currently they are treated as strings. This will cause a problem with > ordering when the id portion of the app id rolls over to the next digit. > For example, "app_1234567890_1" will be considered *earlier* than > "app_1234567890_". We should correct this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[ https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14939315#comment-14939315 ] Joep Rottinghuis commented on YARN-4178: Fine starting without it with benefit of more compact keys. If we add the application_ it should be post fixed in the key to ensure correct ordering. Why do we need any util classes for this, can't an AppId class handle this by itself (convert from string to byte representation and back)? Sent from my iPhone > [storage implementation] app id as string in row keys can cause incorrect > ordering > -- > > Key: YARN-4178 > URL: https://issues.apache.org/jira/browse/YARN-4178 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4178-YARN-2928.01.patch, > YARN-4178-YARN-2928.02.patch > > > Currently the app id is used in various places as part of row keys. However, > currently they are treated as strings. This will cause a problem with > ordering when the id portion of the app id rolls over to the next digit. > For example, "app_1234567890_1" will be considered *earlier* than > "app_1234567890_". We should correct this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[ https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14934954#comment-14934954 ] Varun Saxena commented on YARN-4178: bq. As for the "application" prefix, is there a discussion in the YARN community in general of making this prefix customizable? I'm just not familiar with the current state of that discussion, although I recall hearing about it in the past. That would help us determine whether this is significant enough for us to care about it. I haven't come across any such active discussion recently. [~vinodkv] might be knowing whether there is any such pre-decided future plan as well. Even if a prefix is added in future we would most probably encapsulate that inside ApplicationId class itself i.e. prefix will be a an additional field instead of a static string(application_). So while checking for references of this class, person changing or reviewing the code would come across this code as well and should take necessary action i.e. storing the prefix in HBase, IMO. Thoughts ? > [storage implementation] app id as string in row keys can cause incorrect > ordering > -- > > Key: YARN-4178 > URL: https://issues.apache.org/jira/browse/YARN-4178 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4178-YARN-2928.01.patch > > > Currently the app id is used in various places as part of row keys. However, > currently they are treated as strings. This will cause a problem with > ordering when the id portion of the app id rolls over to the next digit. > For example, "app_1234567890_1" will be considered *earlier* than > "app_1234567890_". We should correct this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[ https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14934934#comment-14934934 ] Varun Saxena commented on YARN-4178: Ok...Will change it. > [storage implementation] app id as string in row keys can cause incorrect > ordering > -- > > Key: YARN-4178 > URL: https://issues.apache.org/jira/browse/YARN-4178 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4178-YARN-2928.01.patch > > > Currently the app id is used in various places as part of row keys. However, > currently they are treated as strings. This will cause a problem with > ordering when the id portion of the app id rolls over to the next digit. > For example, "app_1234567890_1" will be considered *earlier* than > "app_1234567890_". We should correct this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[
https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14935922#comment-14935922
]
Hadoop QA commented on YARN-4178:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 48s | Findbugs (version ) appears to
be broken on YARN-2928. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 53s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 59s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 16s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 41s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 51s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 2m 49s | Tests passed in
hadoop-yarn-server-timelineservice. |
| | | 40m 22s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12764315/YARN-4178-YARN-2928.02.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-2928 / def22b9 |
| hadoop-yarn-server-timelineservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9300/artifact/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9300/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9300/console |
This message was automatically generated.
> [storage implementation] app id as string in row keys can cause incorrect
> ordering
> --
>
> Key: YARN-4178
> URL: https://issues.apache.org/jira/browse/YARN-4178
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
> Attachments: YARN-4178-YARN-2928.01.patch,
> YARN-4178-YARN-2928.02.patch
>
>
> Currently the app id is used in various places as part of row keys. However,
> currently they are treated as strings. This will cause a problem with
> ordering when the id portion of the app id rolls over to the next digit.
> For example, "app_1234567890_1" will be considered *earlier* than
> "app_1234567890_". We should correct this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[ https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14935530#comment-14935530 ] Varun Saxena commented on YARN-4178: That's a good point. Yes, would need to handle it. As you said, should be doable i.e. records with app id stored as 12 bytes can be considered old records and > 12, new ones. > [storage implementation] app id as string in row keys can cause incorrect > ordering > -- > > Key: YARN-4178 > URL: https://issues.apache.org/jira/browse/YARN-4178 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4178-YARN-2928.01.patch > > > Currently the app id is used in various places as part of row keys. However, > currently they are treated as strings. This will cause a problem with > ordering when the id portion of the app id rolls over to the next digit. > For example, "app_1234567890_1" will be considered *earlier* than > "app_1234567890_". We should correct this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[ https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14935531#comment-14935531 ] Varun Saxena commented on YARN-4178: That's a good point. Yes, would need to handle it. As you said, should be doable i.e. records with app id stored as 12 bytes can be considered old records and > 12, new ones. > [storage implementation] app id as string in row keys can cause incorrect > ordering > -- > > Key: YARN-4178 > URL: https://issues.apache.org/jira/browse/YARN-4178 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-4178-YARN-2928.01.patch > > > Currently the app id is used in various places as part of row keys. However, > currently they are treated as strings. This will cause a problem with > ordering when the id portion of the app id rolls over to the next digit. > For example, "app_1234567890_1" will be considered *earlier* than > "app_1234567890_". We should correct this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[
https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14935493#comment-14935493
]
Sangjin Lee commented on YARN-4178:
---
I think the only (minor) concern is if we did not store the prefix
("application") today, and later YARN introduces the custom prefix. Then the
timeline service would need to start storing the new prefix. Then at least we
would need to be able to recognize and handle old records that do not store any
prefix. I suspect that's doable, but am just clarifying the implication
regarding the data schema.
> [storage implementation] app id as string in row keys can cause incorrect
> ordering
> --
>
> Key: YARN-4178
> URL: https://issues.apache.org/jira/browse/YARN-4178
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
> Attachments: YARN-4178-YARN-2928.01.patch
>
>
> Currently the app id is used in various places as part of row keys. However,
> currently they are treated as strings. This will cause a problem with
> ordering when the id portion of the app id rolls over to the next digit.
> For example, "app_1234567890_1" will be considered *earlier* than
> "app_1234567890_". We should correct this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4178) [storage implementation] app id as string in row keys can cause incorrect ordering
[
https://issues.apache.org/jira/browse/YARN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14934442#comment-14934442
]
Sangjin Lee commented on YARN-4178:
---
Thanks [~varun_saxena] for the proposed patch. I'm generally in agreement with
the approach.
As for the "application" prefix, is there a discussion in the YARN community in
general of making this prefix customizable? I'm just not familiar with the
current state of that discussion, although I recall hearing about it in the
past. That would help us determine whether this is significant enough for us to
care about it.
One small nit on the patch: in {{TimelineReaderUtils.java}}, there is following
code:
{code}
long clusterTs = Long.MAX_VALUE - Bytes.toLong(appIdBytes, 0);
int seqId = Integer.MAX_VALUE - Bytes.toInt(appIdBytes, Bytes.SIZEOF_LONG);
{code}
Although it is exactly equivalent, for readability it might be bit better to
spell out the size and the offset explicitly, and also use {{invert()}}:
{code}
long clusterTs = invert(Bytes.toLong(appIdBytes, 0, Bytes.SIZEOF_LONG));
int seqId = invert(Bytes.toInt(appIdBytes, Bytes.SIZEOF_LONG,
Bytes.SIZEOF_INT));
{code}
That would also make it a little more symmetric with the writer util. Also, use
{{invert()}} for the writer util.
> [storage implementation] app id as string in row keys can cause incorrect
> ordering
> --
>
> Key: YARN-4178
> URL: https://issues.apache.org/jira/browse/YARN-4178
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
> Attachments: YARN-4178-YARN-2928.01.patch
>
>
> Currently the app id is used in various places as part of row keys. However,
> currently they are treated as strings. This will cause a problem with
> ordering when the id portion of the app id rolls over to the next digit.
> For example, "app_1234567890_1" will be considered *earlier* than
> "app_1234567890_". We should correct this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
