[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369825#comment-15369825 ] Hudson commented on YARN-4025: -- SUCCESS: Integrated in Hadoop-trunk-Commit #10074 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10074/]) YARN-4025. Deal with byte representations of Longs in writer code. (sjlee: rev 7a41b5501ea76f94f15f53f6380b3c63f14b5a78) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestHBaseTimelineWriterImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/entity/EntityColumnPrefix.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/ColumnHelper.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/application/ApplicationTable.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/TimelineWriterUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseTimelineWriterImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/entity/EntityTable.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseTimelineReaderImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/Separator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/application/ApplicationColumnPrefix.java > Deal with byte representations of Longs in writer code > -- > > Key: YARN-4025 > URL: https://issues.apache.org/jira/browse/YARN-4025 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Sangjin Lee > Fix For: YARN-2928 > > Attachments: YARN-4025-YARN-2928.001.patch, > YARN-4025-YARN-2928.002.patch, YARN-4025-YARN-2928.003.patch, > YARN-4025-YARN-2928.004.patch > > > Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl > code. There seem to be some places in the code where there are conversions > between Long to byte[] to String for easier argument passing between function > calls. Then these values end up being converted back to byte[] while storing > in hbase. > It would be better to pass around byte[] or the Longs themselves as > applicable. > This may result in some api changes (store function) as well in adding a few > more function calls like getColumnQualifier which accepts a pre-encoded byte > array. It will be in addition to the existing api which accepts a String and > the ColumnHelper to return a byte[] column name instead of a String one. > Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14703465#comment-14703465 ] Sangjin Lee commented on YARN-4025: --- Thanks everyone! Deal with byte representations of Longs in writer code -- Key: YARN-4025 URL: https://issues.apache.org/jira/browse/YARN-4025 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Vrushali C Assignee: Sangjin Lee Fix For: YARN-2928 Attachments: YARN-4025-YARN-2928.001.patch, YARN-4025-YARN-2928.002.patch, YARN-4025-YARN-2928.003.patch, YARN-4025-YARN-2928.004.patch Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl code. There seem to be some places in the code where there are conversions between Long to byte[] to String for easier argument passing between function calls. Then these values end up being converted back to byte[] while storing in hbase. It would be better to pass around byte[] or the Longs themselves as applicable. This may result in some api changes (store function) as well in adding a few more function calls like getColumnQualifier which accepts a pre-encoded byte array. It will be in addition to the existing api which accepts a String and the ColumnHelper to return a byte[] column name instead of a String one. Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701656#comment-14701656 ] Junping Du commented on YARN-4025: -- bq. The EntityTable.java file is already fixed in the v.3 patch. I mean the example. id3?id4?id5 should be id3=id4=id5? or I miss something here? :) Deal with byte representations of Longs in writer code -- Key: YARN-4025 URL: https://issues.apache.org/jira/browse/YARN-4025 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Vrushali C Assignee: Sangjin Lee Attachments: YARN-4025-YARN-2928.001.patch, YARN-4025-YARN-2928.002.patch, YARN-4025-YARN-2928.003.patch Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl code. There seem to be some places in the code where there are conversions between Long to byte[] to String for easier argument passing between function calls. Then these values end up being converted back to byte[] while storing in hbase. It would be better to pass around byte[] or the Longs themselves as applicable. This may result in some api changes (store function) as well in adding a few more function calls like getColumnQualifier which accepts a pre-encoded byte array. It will be in addition to the existing api which accepts a String and the ColumnHelper to return a byte[] column name instead of a String one. Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701672#comment-14701672 ] Sangjin Lee commented on YARN-4025: --- Oh OK. Got it. I thought you meant the line you referred to. Deal with byte representations of Longs in writer code -- Key: YARN-4025 URL: https://issues.apache.org/jira/browse/YARN-4025 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Vrushali C Assignee: Sangjin Lee Attachments: YARN-4025-YARN-2928.001.patch, YARN-4025-YARN-2928.002.patch, YARN-4025-YARN-2928.003.patch Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl code. There seem to be some places in the code where there are conversions between Long to byte[] to String for easier argument passing between function calls. Then these values end up being converted back to byte[] while storing in hbase. It would be better to pass around byte[] or the Longs themselves as applicable. This may result in some api changes (store function) as well in adding a few more function calls like getColumnQualifier which accepts a pre-encoded byte array. It will be in addition to the existing api which accepts a String and the ColumnHelper to return a byte[] column name instead of a String one. Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[
https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701645#comment-14701645
]
Sangjin Lee commented on YARN-4025:
---
Thanks for your review [~djp]!
{quote}
Do we handle columnPrefixBytes to be null here as Javadoc comments? I saw we
handle this case explicitly in readResults() but I didn't see it here. Let me
know if I miss something.
{quote}
That is a good catch. Let me look into that. If we retain the same behavior for
a null qualifier (and probably we should), then the return type of this method
would need to go back to {{MapObject, Object}}. I'll also think about the
method names. Cc'ing [~vrushalic] for her opinion also.
{quote}
Checking with javadoc in Separator and TimelineWriterUtils - a negative value
indicates no limit on number of segments., so can we define a constant value
like NO_LIMIT to replace -1 here?
{quote}
Will do.
{quote}
I think we should do the same thing to some javadoc examples in
EntityTable.java.
{quote}
The {{EntityTable.java}} file is already fixed in the v.3 patch.
{quote}
Forget to mention that, YARN-3049 should rename TestHBaseTimelineWriterImpl to
something include Reader. Would you like to do it here? Thanks!
{quote}
Good idea. The thought definitely occurred to me.
I'll update the patch pretty soon.
Deal with byte representations of Longs in writer code
--
Key: YARN-4025
URL: https://issues.apache.org/jira/browse/YARN-4025
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Vrushali C
Assignee: Sangjin Lee
Attachments: YARN-4025-YARN-2928.001.patch,
YARN-4025-YARN-2928.002.patch, YARN-4025-YARN-2928.003.patch
Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl
code. There seem to be some places in the code where there are conversions
between Long to byte[] to String for easier argument passing between function
calls. Then these values end up being converted back to byte[] while storing
in hbase.
It would be better to pass around byte[] or the Longs themselves as
applicable.
This may result in some api changes (store function) as well in adding a few
more function calls like getColumnQualifier which accepts a pre-encoded byte
array. It will be in addition to the existing api which accepts a String and
the ColumnHelper to return a byte[] column name instead of a String one.
Filing jira to track these changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[
https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14702212#comment-14702212
]
Hadoop QA commented on YARN-4025:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 58s | Findbugs (version ) appears to
be broken on YARN-2928. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 8m 35s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 44s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 18s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 5s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 42s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 52s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 1m 29s | Tests passed in
hadoop-yarn-server-timelineservice. |
| | | 41m 45s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12751118/YARN-4025-YARN-2928.004.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-2928 / 9a82008 |
| hadoop-yarn-server-timelineservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8879/artifact/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8879/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8879/console |
This message was automatically generated.
Deal with byte representations of Longs in writer code
--
Key: YARN-4025
URL: https://issues.apache.org/jira/browse/YARN-4025
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Vrushali C
Assignee: Sangjin Lee
Attachments: YARN-4025-YARN-2928.001.patch,
YARN-4025-YARN-2928.002.patch, YARN-4025-YARN-2928.003.patch,
YARN-4025-YARN-2928.004.patch
Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl
code. There seem to be some places in the code where there are conversions
between Long to byte[] to String for easier argument passing between function
calls. Then these values end up being converted back to byte[] while storing
in hbase.
It would be better to pass around byte[] or the Longs themselves as
applicable.
This may result in some api changes (store function) as well in adding a few
more function calls like getColumnQualifier which accepts a pre-encoded byte
array. It will be in addition to the existing api which accepts a String and
the ColumnHelper to return a byte[] column name instead of a String one.
Filing jira to track these changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14702274#comment-14702274 ] Junping Du commented on YARN-4025: -- bq. We can treat that method as the default readResults() method, and the other one as the one for having raw (non-string) components. I added more javadoc to clarify that point. Sounds good. Thanks for addressing this and other review comments. +1 on latest (004) patch. Will commit it shortly if no further comments from others. Deal with byte representations of Longs in writer code -- Key: YARN-4025 URL: https://issues.apache.org/jira/browse/YARN-4025 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Vrushali C Assignee: Sangjin Lee Attachments: YARN-4025-YARN-2928.001.patch, YARN-4025-YARN-2928.002.patch, YARN-4025-YARN-2928.003.patch, YARN-4025-YARN-2928.004.patch Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl code. There seem to be some places in the code where there are conversions between Long to byte[] to String for easier argument passing between function calls. Then these values end up being converted back to byte[] while storing in hbase. It would be better to pass around byte[] or the Longs themselves as applicable. This may result in some api changes (store function) as well in adding a few more function calls like getColumnQualifier which accepts a pre-encoded byte array. It will be in addition to the existing api which accepts a String and the ColumnHelper to return a byte[] column name instead of a String one. Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701325#comment-14701325 ] Junping Du commented on YARN-4025: -- bq. One major change I did is that now TestHBaseTimelineWriterImpl verifies the timeline entities read by HBaseTimelineReaderImpl as well. This provides a nice benefit of verifying correctness of HBaseTimelineReaderImpl. It uncovered a bug in the process. Nice work! Forget to mention that, YARN-3049 should rename TestHBaseTimelineWriterImpl to something include Reader. Would you like to do it here? Thanks! Deal with byte representations of Longs in writer code -- Key: YARN-4025 URL: https://issues.apache.org/jira/browse/YARN-4025 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Vrushali C Assignee: Sangjin Lee Attachments: YARN-4025-YARN-2928.001.patch, YARN-4025-YARN-2928.002.patch, YARN-4025-YARN-2928.003.patch Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl code. There seem to be some places in the code where there are conversions between Long to byte[] to String for easier argument passing between function calls. Then these values end up being converted back to byte[] while storing in hbase. It would be better to pass around byte[] or the Longs themselves as applicable. This may result in some api changes (store function) as well in adding a few more function calls like getColumnQualifier which accepts a pre-encoded byte array. It will be in addition to the existing api which accepts a String and the ColumnHelper to return a byte[] column name instead of a String one. Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[
https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701309#comment-14701309
]
Junping Du commented on YARN-4025:
--
Thanks [~sjlee0] for updating the patch. 003 patch looks good to me in general.
Some minor comments:
In ColumnHelper.java,
{code}
/**
...
+ * @param columnPrefixBytes
+ * optional prefix to limit columns. If null all columns are
+ * returned.
...
+ */
+ public Mapbyte[][], Object readResultsHavingCompoundColumnQualifiers(
{code}
Do we handle columnPrefixBytes to be null here as Javadoc comments? I saw we
handle this case explicitly in readResults() but I didn't see it here. Let me
know if I miss something. In addition, looks like previous readResults() only
handle the case CQs are all String. I think we should update that method name
to something like: readResultsWithAllStringColumnQualifiers() to get rid of
possible confusion. Last but not the least, for result to be null case, do we
need to handle it with log some warn messages like other cases in this patch?
{code}
+ byte[][] columnQualifierParts = Separator.VALUES.split(
+ columnNameParts[1], -1);
{code}
Checking with javadoc in Separator and TimelineWriterUtils - a negative value
indicates no limit on number of segments., so can we define a constant value
like NO_LIMIT to replace -1 here? Actually, from checking with implementation
in TimelineWriterUtils, 0 also indicates the same thing (no limit). Sounds like
we don't have any tests in TestTimelineWriterUtils.java, we may want to improve
this in future?
In ApplicationTable,
{code}
- * || e!eventId?timestamp?infoKey: | | |
+ * || e!eventId=timestamp=infoKey: |
{code}
I think we should do the same thing to some javadoc examples in
EntityTable.java.
Other looks fine to me.
Deal with byte representations of Longs in writer code
--
Key: YARN-4025
URL: https://issues.apache.org/jira/browse/YARN-4025
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Vrushali C
Assignee: Sangjin Lee
Attachments: YARN-4025-YARN-2928.001.patch,
YARN-4025-YARN-2928.002.patch, YARN-4025-YARN-2928.003.patch
Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl
code. There seem to be some places in the code where there are conversions
between Long to byte[] to String for easier argument passing between function
calls. Then these values end up being converted back to byte[] while storing
in hbase.
It would be better to pass around byte[] or the Longs themselves as
applicable.
This may result in some api changes (store function) as well in adding a few
more function calls like getColumnQualifier which accepts a pre-encoded byte
array. It will be in addition to the existing api which accepts a String and
the ColumnHelper to return a byte[] column name instead of a String one.
Filing jira to track these changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14700524#comment-14700524 ] Li Lu commented on YARN-4025: - Latest patch LGTM. Thanks [~sjlee0]! Deal with byte representations of Longs in writer code -- Key: YARN-4025 URL: https://issues.apache.org/jira/browse/YARN-4025 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Vrushali C Assignee: Sangjin Lee Attachments: YARN-4025-YARN-2928.001.patch, YARN-4025-YARN-2928.002.patch, YARN-4025-YARN-2928.003.patch Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl code. There seem to be some places in the code where there are conversions between Long to byte[] to String for easier argument passing between function calls. Then these values end up being converted back to byte[] while storing in hbase. It would be better to pass around byte[] or the Longs themselves as applicable. This may result in some api changes (store function) as well in adding a few more function calls like getColumnQualifier which accepts a pre-encoded byte array. It will be in addition to the existing api which accepts a String and the ColumnHelper to return a byte[] column name instead of a String one. Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[
https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14700628#comment-14700628
]
Hadoop QA commented on YARN-4025:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 25s | Findbugs (version ) appears to
be broken on YARN-2928. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 36s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 29s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 17s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 6s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 43s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 55s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 1m 27s | Tests passed in
hadoop-yarn-server-timelineservice. |
| | | 41m 57s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12750907/YARN-4025-YARN-2928.003.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-2928 / a029ce1 |
| hadoop-yarn-server-timelineservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8870/artifact/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8870/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8870/console |
This message was automatically generated.
Deal with byte representations of Longs in writer code
--
Key: YARN-4025
URL: https://issues.apache.org/jira/browse/YARN-4025
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Vrushali C
Assignee: Sangjin Lee
Attachments: YARN-4025-YARN-2928.001.patch,
YARN-4025-YARN-2928.002.patch, YARN-4025-YARN-2928.003.patch
Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl
code. There seem to be some places in the code where there are conversions
between Long to byte[] to String for easier argument passing between function
calls. Then these values end up being converted back to byte[] while storing
in hbase.
It would be better to pass around byte[] or the Longs themselves as
applicable.
This may result in some api changes (store function) as well in adding a few
more function calls like getColumnQualifier which accepts a pre-encoded byte
array. It will be in addition to the existing api which accepts a String and
the ColumnHelper to return a byte[] column name instead of a String one.
Filing jira to track these changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1472#comment-1472 ] Sangjin Lee commented on YARN-4025: --- That's a good point [~gtCarrera9]. Let me see if I can improve on that point. Deal with byte representations of Longs in writer code -- Key: YARN-4025 URL: https://issues.apache.org/jira/browse/YARN-4025 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Vrushali C Assignee: Sangjin Lee Attachments: YARN-4025-YARN-2928.001.patch, YARN-4025-YARN-2928.002.patch Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl code. There seem to be some places in the code where there are conversions between Long to byte[] to String for easier argument passing between function calls. Then these values end up being converted back to byte[] while storing in hbase. It would be better to pass around byte[] or the Longs themselves as applicable. This may result in some api changes (store function) as well in adding a few more function calls like getColumnQualifier which accepts a pre-encoded byte array. It will be in addition to the existing api which accepts a String and the ColumnHelper to return a byte[] column name instead of a String one. Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697531#comment-14697531 ] Vrushali C commented on YARN-4025: -- Hmm, yes I think some more comments there might help (I should have included them in the earlier patch) Deal with byte representations of Longs in writer code -- Key: YARN-4025 URL: https://issues.apache.org/jira/browse/YARN-4025 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Vrushali C Assignee: Sangjin Lee Attachments: YARN-4025-YARN-2928.001.patch, YARN-4025-YARN-2928.002.patch Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl code. There seem to be some places in the code where there are conversions between Long to byte[] to String for easier argument passing between function calls. Then these values end up being converted back to byte[] while storing in hbase. It would be better to pass around byte[] or the Longs themselves as applicable. This may result in some api changes (store function) as well in adding a few more function calls like getColumnQualifier which accepts a pre-encoded byte array. It will be in addition to the existing api which accepts a String and the ColumnHelper to return a byte[] column name instead of a String one. Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697534#comment-14697534 ] Vrushali C commented on YARN-4025: -- I changed it from '?' to '='. Sangjin was also wondering if we should change it or no (read earlier comments in the jira). I think it might be good to change it now since '?' is a wild card character and using a non wild card character helps in easier reading while testing and for the reader code as well. It's a very small change, so thought another jira for this was an overkill. Deal with byte representations of Longs in writer code -- Key: YARN-4025 URL: https://issues.apache.org/jira/browse/YARN-4025 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Vrushali C Assignee: Sangjin Lee Attachments: YARN-4025-YARN-2928.001.patch, YARN-4025-YARN-2928.002.patch Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl code. There seem to be some places in the code where there are conversions between Long to byte[] to String for easier argument passing between function calls. Then these values end up being converted back to byte[] while storing in hbase. It would be better to pass around byte[] or the Longs themselves as applicable. This may result in some api changes (store function) as well in adding a few more function calls like getColumnQualifier which accepts a pre-encoded byte array. It will be in addition to the existing api which accepts a String and the ColumnHelper to return a byte[] column name instead of a String one. Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697545#comment-14697545 ] Li Lu commented on YARN-4025: - Oh sorry I missed that line... That looks fine. Deal with byte representations of Longs in writer code -- Key: YARN-4025 URL: https://issues.apache.org/jira/browse/YARN-4025 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Vrushali C Assignee: Sangjin Lee Attachments: YARN-4025-YARN-2928.001.patch, YARN-4025-YARN-2928.002.patch Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl code. There seem to be some places in the code where there are conversions between Long to byte[] to String for easier argument passing between function calls. Then these values end up being converted back to byte[] while storing in hbase. It would be better to pass around byte[] or the Longs themselves as applicable. This may result in some api changes (store function) as well in adding a few more function calls like getColumnQualifier which accepts a pre-encoded byte array. It will be in addition to the existing api which accepts a String and the ColumnHelper to return a byte[] column name instead of a String one. Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[
https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694840#comment-14694840
]
Hadoop QA commented on YARN-4025:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 20m 36s | Findbugs (version ) appears to
be broken on YARN-2928. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 26s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 24s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 18s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 4s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 44s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 50s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 52s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 1m 26s | Tests passed in
hadoop-yarn-server-timelineservice. |
| | | 45m 15s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12750226/YARN-4025-YARN-2928.002.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-2928 / bcd755e |
| hadoop-yarn-server-timelineservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8839/artifact/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8839/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8839/console |
This message was automatically generated.
Deal with byte representations of Longs in writer code
--
Key: YARN-4025
URL: https://issues.apache.org/jira/browse/YARN-4025
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Vrushali C
Assignee: Sangjin Lee
Attachments: YARN-4025-YARN-2928.001.patch,
YARN-4025-YARN-2928.002.patch
Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl
code. There seem to be some places in the code where there are conversions
between Long to byte[] to String for easier argument passing between function
calls. Then these values end up being converted back to byte[] while storing
in hbase.
It would be better to pass around byte[] or the Longs themselves as
applicable.
This may result in some api changes (store function) as well in adding a few
more function calls like getColumnQualifier which accepts a pre-encoded byte
array. It will be in addition to the existing api which accepts a String and
the ColumnHelper to return a byte[] column name instead of a String one.
Filing jira to track these changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14696268#comment-14696268 ] Li Lu commented on YARN-4025: - Hi [~sjlee0], thanks for the work! The patch overall LGTM. One small question: The way to deserializing event key, ts, and infokey appears to be a little bit isolated from the serialization part in the writer. Maybe it's helpful to put them in a centralized place? It might be a little bit confusing for the hardcoded 0, 1, and 2 indices, so maybe it's also helpful to mark down the structure of the entityid:ts:infokey structure somewhere close? Deal with byte representations of Longs in writer code -- Key: YARN-4025 URL: https://issues.apache.org/jira/browse/YARN-4025 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Vrushali C Assignee: Sangjin Lee Attachments: YARN-4025-YARN-2928.001.patch, YARN-4025-YARN-2928.002.patch Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl code. There seem to be some places in the code where there are conversions between Long to byte[] to String for easier argument passing between function calls. Then these values end up being converted back to byte[] while storing in hbase. It would be better to pass around byte[] or the Longs themselves as applicable. This may result in some api changes (store function) as well in adding a few more function calls like getColumnQualifier which accepts a pre-encoded byte array. It will be in addition to the existing api which accepts a String and the ColumnHelper to return a byte[] column name instead of a String one. Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14696271#comment-14696271 ] Li Lu commented on YARN-4025: - Oh and, BTW, why are we changing Separator.VALUES? Deal with byte representations of Longs in writer code -- Key: YARN-4025 URL: https://issues.apache.org/jira/browse/YARN-4025 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Vrushali C Assignee: Sangjin Lee Attachments: YARN-4025-YARN-2928.001.patch, YARN-4025-YARN-2928.002.patch Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl code. There seem to be some places in the code where there are conversions between Long to byte[] to String for easier argument passing between function calls. Then these values end up being converted back to byte[] while storing in hbase. It would be better to pass around byte[] or the Longs themselves as applicable. This may result in some api changes (store function) as well in adding a few more function calls like getColumnQualifier which accepts a pre-encoded byte array. It will be in addition to the existing api which accepts a String and the ColumnHelper to return a byte[] column name instead of a String one. Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694177#comment-14694177 ] Vrushali C commented on YARN-4025: -- Reassigning to Sangjin Deal with byte representations of Longs in writer code -- Key: YARN-4025 URL: https://issues.apache.org/jira/browse/YARN-4025 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Vrushali C Assignee: Sangjin Lee Attachments: YARN-4025-YARN-2928.001.patch Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl code. There seem to be some places in the code where there are conversions between Long to byte[] to String for easier argument passing between function calls. Then these values end up being converted back to byte[] while storing in hbase. It would be better to pass around byte[] or the Longs themselves as applicable. This may result in some api changes (store function) as well in adding a few more function calls like getColumnQualifier which accepts a pre-encoded byte array. It will be in addition to the existing api which accepts a String and the ColumnHelper to return a byte[] column name instead of a String one. Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14692511#comment-14692511 ] Vrushali C commented on YARN-4025: -- Yes, +1 Deal with byte representations of Longs in writer code -- Key: YARN-4025 URL: https://issues.apache.org/jira/browse/YARN-4025 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Vrushali C Assignee: Vrushali C Attachments: YARN-4025-YARN-2928.001.patch Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl code. There seem to be some places in the code where there are conversions between Long to byte[] to String for easier argument passing between function calls. Then these values end up being converted back to byte[] while storing in hbase. It would be better to pass around byte[] or the Longs themselves as applicable. This may result in some api changes (store function) as well in adding a few more function calls like getColumnQualifier which accepts a pre-encoded byte array. It will be in addition to the existing api which accepts a String and the ColumnHelper to return a byte[] column name instead of a String one. Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14692468#comment-14692468 ] Sangjin Lee commented on YARN-4025: --- For the record, we will go ahead with YARN-3906 first. We'll need to update this patch to reflect the changes in YARN-3906. I'll work with [~vrushalic] on that. Deal with byte representations of Longs in writer code -- Key: YARN-4025 URL: https://issues.apache.org/jira/browse/YARN-4025 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Vrushali C Assignee: Vrushali C Attachments: YARN-4025-YARN-2928.001.patch Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl code. There seem to be some places in the code where there are conversions between Long to byte[] to String for easier argument passing between function calls. Then these values end up being converted back to byte[] while storing in hbase. It would be better to pass around byte[] or the Longs themselves as applicable. This may result in some api changes (store function) as well in adding a few more function calls like getColumnQualifier which accepts a pre-encoded byte array. It will be in addition to the existing api which accepts a String and the ColumnHelper to return a byte[] column name instead of a String one. Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680393#comment-14680393 ] Li Lu commented on YARN-4025: - Hi [~sjlee0][~vrushalic], I slightly prefer to firstly move with YARN-3906, since it's a much bigger patch to refresh. However I'm OK with either ways. Deal with byte representations of Longs in writer code -- Key: YARN-4025 URL: https://issues.apache.org/jira/browse/YARN-4025 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Vrushali C Assignee: Vrushali C Attachments: YARN-4025-YARN-2928.001.patch Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl code. There seem to be some places in the code where there are conversions between Long to byte[] to String for easier argument passing between function calls. Then these values end up being converted back to byte[] while storing in hbase. It would be better to pass around byte[] or the Longs themselves as applicable. This may result in some api changes (store function) as well in adding a few more function calls like getColumnQualifier which accepts a pre-encoded byte array. It will be in addition to the existing api which accepts a String and the ColumnHelper to return a byte[] column name instead of a String one. Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680388#comment-14680388 ] Sangjin Lee commented on YARN-4025: --- Thanks for the patch [~vrushalic]. I believe you need to rebase with the latest from YARN-2928 (especially the last commit for YARN-3049). This patch and YARN-3906 which I'm working on have several files that are overlapping, and we need to decide the order in which these patches go in. I'm open to either order. Thoughts? Deal with byte representations of Longs in writer code -- Key: YARN-4025 URL: https://issues.apache.org/jira/browse/YARN-4025 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Vrushali C Assignee: Vrushali C Attachments: YARN-4025-YARN-2928.001.patch Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl code. There seem to be some places in the code where there are conversions between Long to byte[] to String for easier argument passing between function calls. Then these values end up being converted back to byte[] while storing in hbase. It would be better to pass around byte[] or the Longs themselves as applicable. This may result in some api changes (store function) as well in adding a few more function calls like getColumnQualifier which accepts a pre-encoded byte array. It will be in addition to the existing api which accepts a String and the ColumnHelper to return a byte[] column name instead of a String one. Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680450#comment-14680450 ] Sangjin Lee commented on YARN-4025: --- No, not really. I just wanted to understand the situation. Since we're properly escaping both ways, we're probably ok with either of them, as long as they rarely appear in the actual values themselves. Deal with byte representations of Longs in writer code -- Key: YARN-4025 URL: https://issues.apache.org/jira/browse/YARN-4025 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Vrushali C Assignee: Vrushali C Attachments: YARN-4025-YARN-2928.001.patch Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl code. There seem to be some places in the code where there are conversions between Long to byte[] to String for easier argument passing between function calls. Then these values end up being converted back to byte[] while storing in hbase. It would be better to pass around byte[] or the Longs themselves as applicable. This may result in some api changes (store function) as well in adding a few more function calls like getColumnQualifier which accepts a pre-encoded byte array. It will be in addition to the existing api which accepts a String and the ColumnHelper to return a byte[] column name instead of a String one. Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[
https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680410#comment-14680410
]
Sangjin Lee commented on YARN-4025:
---
Went over the patch pretty quickly, and have some high level comments.
- now that YARN-3049 is in, you'd need to modify {{HBaseTimelineReaderImpl}}
also to do the right thing regarding the timestamps.
- I am making a similar change for refactoring the unit tests in YARN-3906. I'm
also renaming the methods to make it more explicit what the test is about
(please see the latest patch for YARN-3906). So depending on the order in which
these changes go in, we need to reconcile this. Just a heads up.
- Regarding changing the character for the separator, is it causing a real bug?
Do we really need to change it? Note that we want to update the table
description javadoc too if we're going to change it.
Deal with byte representations of Longs in writer code
--
Key: YARN-4025
URL: https://issues.apache.org/jira/browse/YARN-4025
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Vrushali C
Assignee: Vrushali C
Attachments: YARN-4025-YARN-2928.001.patch
Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl
code. There seem to be some places in the code where there are conversions
between Long to byte[] to String for easier argument passing between function
calls. Then these values end up being converted back to byte[] while storing
in hbase.
It would be better to pass around byte[] or the Longs themselves as
applicable.
This may result in some api changes (store function) as well in adding a few
more function calls like getColumnQualifier which accepts a pre-encoded byte
array. It will be in addition to the existing api which accepts a String and
the ColumnHelper to return a byte[] column name instead of a String one.
Filing jira to track these changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680430#comment-14680430 ] Vrushali C commented on YARN-4025: -- Thanks [~sjlee0] and [~gtCarrera9] I will rebase and generate a new patch. bq. Regarding changing the character for the separator, is it causing a real bug? Do we really need to change it? Note that we want to update the table description javadoc too if we're going to change it. I don't think it will cause a bug if used properly while reading. I think it may be better to use something that is not a wildcard character like * or ?. I can update the javadoc. Is there any concern with changing the separator? Deal with byte representations of Longs in writer code -- Key: YARN-4025 URL: https://issues.apache.org/jira/browse/YARN-4025 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Vrushali C Assignee: Vrushali C Attachments: YARN-4025-YARN-2928.001.patch Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl code. There seem to be some places in the code where there are conversions between Long to byte[] to String for easier argument passing between function calls. Then these values end up being converted back to byte[] while storing in hbase. It would be better to pass around byte[] or the Longs themselves as applicable. This may result in some api changes (store function) as well in adding a few more function calls like getColumnQualifier which accepts a pre-encoded byte array. It will be in addition to the existing api which accepts a String and the ColumnHelper to return a byte[] column name instead of a String one. Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[
https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662833#comment-14662833
]
Vrushali C commented on YARN-4025:
--
The function {code} public MapString, Object readResults(Result result,
byte[] columnPrefixBytes) {code} needs to be updated to read the column
qualifier as a separator separated list of byte arrays instead of strings. That
means the Separator class also needs to split the inputs as byte arrays not as
Strings. This is to enable dealing with Long values in column qualifiers.
Deal with byte representations of Longs in writer code
--
Key: YARN-4025
URL: https://issues.apache.org/jira/browse/YARN-4025
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Vrushali C
Assignee: Vrushali C
Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl
code. There seem to be some places in the code where there are conversions
between Long to byte[] to String for easier argument passing between function
calls. Then these values end up being converted back to byte[] while storing
in hbase.
It would be better to pass around byte[] or the Longs themselves as
applicable.
This may result in some api changes (store function) as well in adding a few
more function calls like getColumnQualifier which accepts a pre-encoded byte
array. It will be in addition to the existing api which accepts a String and
the ColumnHelper to return a byte[] column name instead of a String one.
Filing jira to track these changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4025) Deal with byte representations of Longs in writer code
[
https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662869#comment-14662869
]
Hadoop QA commented on YARN-4025:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12749410/YARN-4025-YARN-2928.001.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-2928 / 07433c2 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8799/console |
This message was automatically generated.
Deal with byte representations of Longs in writer code
--
Key: YARN-4025
URL: https://issues.apache.org/jira/browse/YARN-4025
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Vrushali C
Assignee: Vrushali C
Attachments: YARN-4025-YARN-2928.001.patch
Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl
code. There seem to be some places in the code where there are conversions
between Long to byte[] to String for easier argument passing between function
calls. Then these values end up being converted back to byte[] while storing
in hbase.
It would be better to pass around byte[] or the Longs themselves as
applicable.
This may result in some api changes (store function) as well in adding a few
more function calls like getColumnQualifier which accepts a pre-encoded byte
array. It will be in addition to the existing api which accepts a String and
the ColumnHelper to return a byte[] column name instead of a String one.
Filing jira to track these changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
