[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524924#comment-14524924
]
Hadoop QA commented on HDFS-5274:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12660003/HDFS-5274-17.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle site |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10658/console |
This message was automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: 3node_get_200mb.png, 3node_put_200mb.png,
3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch,
HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch,
HDFS-5274-14.patch, HDFS-5274-15.patch, HDFS-5274-16.patch,
HDFS-5274-17.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch,
HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch,
HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png,
Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524957#comment-14524957
]
Hadoop QA commented on HDFS-5274:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12660003/HDFS-5274-17.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle site |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10669/console |
This message was automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: 3node_get_200mb.png, 3node_put_200mb.png,
3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch,
HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch,
HDFS-5274-14.patch, HDFS-5274-15.patch, HDFS-5274-16.patch,
HDFS-5274-17.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch,
HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch,
HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png,
Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14103159#comment-14103159 ] Masatake Iwasaki commented on HDFS-5274: I am going to split the patch into 3 pieces. 1. Adding tracing to Hadoop RPC 2. Adding tracing to DataNode data transfer protocol 3. Adding tracing spans to HDFS #2 depends on #1 and #3 depends on #2. Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: 3node_get_200mb.png, 3node_put_200mb.png, 3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch, HDFS-5274-14.patch, HDFS-5274-15.patch, HDFS-5274-16.patch, HDFS-5274-17.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14103163#comment-14103163 ] Nick Dimiduk commented on HDFS-5274: thanks for sticking with this one [~iwasakims]! Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: 3node_get_200mb.png, 3node_put_200mb.png, 3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch, HDFS-5274-14.patch, HDFS-5274-15.patch, HDFS-5274-16.patch, HDFS-5274-17.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087305#comment-14087305
]
Hadoop QA commented on HDFS-5274:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660003/HDFS-5274-17.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.server.namenode.ha.TestDFSZKFailoverController
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7566//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7566//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7566//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: 3node_get_200mb.png, 3node_put_200mb.png,
3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch,
HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch,
HDFS-5274-14.patch, HDFS-5274-15.patch, HDFS-5274-16.patch,
HDFS-5274-17.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch,
HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch,
HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png,
Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087021#comment-14087021 ] Masatake Iwasaki commented on HDFS-5274: By the way, should I split the patch for ease of review? The patch could be split in two as # common part such as fixes in RPC and span receivers, # codes for tracing of HDFS logic. Once the former is merged, developers can write their own (temporary) tracing code for developing features. I appreciate the comments from reviewers and committers. Thanks. Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: 3node_get_200mb.png, 3node_put_200mb.png, 3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch, HDFS-5274-14.patch, HDFS-5274-15.patch, HDFS-5274-16.patch, HDFS-5274-17.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13971846#comment-13971846 ] Elliott Clark commented on HDFS-5274: - Please don't add htrace-zpikin as a dependency. That version of thrift pulled in is very old and we don't want to have classpath issues because of a new feature. Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: 3node_get_200mb.png, 3node_put_200mb.png, 3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch, HDFS-5274-14.patch, HDFS-5274-15.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13972113#comment-13972113 ] Masatake Iwasaki commented on HDFS-5274: attaching updated patch. - rebased - removed htrace-zipkin from pom.xml Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: 3node_get_200mb.png, 3node_put_200mb.png, 3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch, HDFS-5274-14.patch, HDFS-5274-15.patch, HDFS-5274-16.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13972253#comment-13972253
]
Hadoop QA commented on HDFS-5274:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12640555/HDFS-5274-16.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6680//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6680//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: 3node_get_200mb.png, 3node_put_200mb.png,
3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch,
HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch,
HDFS-5274-14.patch, HDFS-5274-15.patch, HDFS-5274-16.patch,
HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch,
HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch,
HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace
d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13956409#comment-13956409
]
Hadoop QA commented on HDFS-5274:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12638024/HDFS-5274-15.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6566//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6566//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: 3node_get_200mb.png, 3node_put_200mb.png,
3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch,
HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch,
HDFS-5274-14.patch, HDFS-5274-15.patch, HDFS-5274-2.patch, HDFS-5274-3.patch,
HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch,
HDFS-5274-8.patch, HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace
a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png,
ss-5274v8-put.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955093#comment-13955093
]
Hadoop QA commented on HDFS-5274:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12637769/HDFS-5274-14.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6560//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6560//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: 3node_get_200mb.png, 3node_put_200mb.png,
3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch,
HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch,
HDFS-5274-14.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch,
HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch,
HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png,
Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13932436#comment-13932436 ] stack commented on HDFS-5274: - [~iwasakims] That is a beautiful png Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: 3node_get_200mb.png, 3node_put_200mb.png, 3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13930756#comment-13930756
]
Hadoop QA commented on HDFS-5274:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12633965/3node_put_200mb.png
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6373//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: 3node_get_200mb.png, 3node_put_200mb.png,
3node_put_200mb.png, HDFS-5274-0.patch, HDFS-5274-1.patch,
HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch,
HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch,
HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch,
HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace
d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13924490#comment-13924490
]
Hadoop QA commented on HDFS-5274:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12633485/3node_get_200mb.png
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6349//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: 3node_get_200mb.png, 3node_put_200mb.png,
HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-10.patch, HDFS-5274-11.txt,
HDFS-5274-12.patch, HDFS-5274-13.patch, HDFS-5274-2.patch, HDFS-5274-3.patch,
HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch,
HDFS-5274-8.patch, HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace
a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png,
ss-5274v8-put.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920058#comment-13920058
]
Hadoop QA commented on HDFS-5274:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12632559/HDFS-5274-12.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6297//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6297//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch,
HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch,
HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch,
HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch,
HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace
d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920078#comment-13920078
]
Hadoop QA commented on HDFS-5274:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12632567/HDFS-5274-13.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.cli.TestAclCLI
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6298//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6298//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch,
HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-12.patch, HDFS-5274-13.patch,
HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch,
HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch,
HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace
d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13918714#comment-13918714
]
Hadoop QA commented on HDFS-5274:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12632352/HDFS-5274-11.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 6 new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.tracing.TestTracing
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6281//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6281//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6281//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6281//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch,
HDFS-5274-10.patch, HDFS-5274-11.txt, HDFS-5274-2.patch, HDFS-5274-3.patch,
HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch,
HDFS-5274-8.patch, HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace
a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png,
ss-5274v8-put.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13917814#comment-13917814
]
Hadoop QA commented on HDFS-5274:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12632181/HDFS-5274-10.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
hadoop-hdfs-project/hadoop-hdfs
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6274//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6274//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6274//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6274//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch,
HDFS-5274-10.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch,
HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch,
HDFS-5274-8.patch, HDFS-5274-9.patch, Zipkin Trace a06e941b0172ec73.png,
Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913301#comment-13913301 ] Masatake Iwasaki commented on HDFS-5274: Hi [~stack], it worked! Please make a release. I will attach a rebased patch later. Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913516#comment-13913516 ] stack commented on HDFS-5274: - Hmm... Add this repository in the top level hadoop pom? +repository + idoss.sonatype/id + nameSonatype OSS Repository/name + urlhttps://oss.sonatype.org/content/groups/public/url + snapshots +enabledtrue/enabled + /snapshots + releases +enabledtrue/enabled + /releases +/repository That makes it work for me. Thanks [~iwasakims] Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913853#comment-13913853
]
Hadoop QA commented on HDFS-5274:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631366/HDFS-5274-9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
hadoop-hdfs-project/hadoop-hdfs
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6249//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6249//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch,
HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch,
HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch, HDFS-5274-9.patch,
Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png,
ss-5274v8-get.png, ss-5274v8-put.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13912252#comment-13912252 ] stack commented on HDFS-5274: - I put up a SNAPSHOT of the tracing jar w/ its groupId changed from org.cloudera.htrace to org.htrace. The version is currently 3.0-SNAPSHOT (all is the same except for the redone domain). If it works for you [~iwasakims], I'll make a release (if the former owners of this jar are good w/ my manglings). Thanks boss. Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13912367#comment-13912367 ] Masatake Iwasaki commented on HDFS-5274: Thanks [~stack]! It seems not yet to be visible in repo.. I will check again a few hours later. Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13908918#comment-13908918 ] stack commented on HDFS-5274: - bq. But I am afraid this could start a trend of capabilities hosted/attributed especially to vendor companies making its way into Hadoop. [~sureshms] I just got perms on this lib. I'll go get a domain, convert the code package names, make a new release, and publish to maven central... will report back when done. Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13908933#comment-13908933 ] Suresh Srinivas commented on HDFS-5274: --- Thanks [~stack]! Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13909065#comment-13909065
]
Hadoop QA commented on HDFS-5274:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12630393/HDFS-5274-8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.web.TestWebHDFS
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6208//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6208//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch,
HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch,
HDFS-5274-7.patch, HDFS-5274-8.patch, HDFS-5274-8.patch, Zipkin Trace
a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png,
ss-5274v8-put.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13907393#comment-13907393 ] Masatake Iwasaki commented on HDFS-5274: Hi [~stack], thanks for your review comments. bq. Have you tried it outside of the unit tests to make sure you get sensible looking spans and numbers? I checked the trace of putting and getting a big file by Zipkin today. There seems to be too many spans concerning DFSInputStream.read and DFSOutputStream.write. I will fix this in the next version of patch. Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13907524#comment-13907524 ] Suresh Srinivas commented on HDFS-5274: --- Is there any plan on making HTrace libraries org.cloudera.htrace.* available in Hadoop common? Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13907800#comment-13907800 ] Todd Lipcon commented on HDFS-5274: --- Hi Suresh. As one of the primary authors of HTrace I'd say that there are no plans to put it in Hadoop Common. Personally I think Hadoop Common as a grab-bag of all things Hadoop gets pretty messy, since it's harder to piece-meal upgrade different components. It also means that dependent apps like HBase would need to wait on new versions of Common and have even more difficulty building the same code against different versions of Hadoop. Happy to take contributions to HTrace via github pull request if you like, though. Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13907816#comment-13907816 ] stack commented on HDFS-5274: - [~iwasakims] Thanks. When you are done, I'll try hooking it up w/ hbase to make sure we get a trace that spans the two systems. [~sureshms] Wouldn't adding htrace to the common pom.xml make it ...available in Hadoop common? Thanks. Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13907917#comment-13907917 ] Suresh Srinivas commented on HDFS-5274: --- bq. Wouldn't adding htrace to the common pom.xml make it ...available in Hadoop common? Thanks. I agree with the comment [~cutting] had made - https://issues.apache.org/jira/browse/HADOOP-10311?focusedCommentId=13886809page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13886809. I would love to see the code in Hadoop community itself, if possible. Agreed, this is not different using google guava or other such libraries. But I am afraid this could start a trend of capabilities hosted/attributed especially to vendor companies making its way into Hadoop. With that said, I am -0 and would rather not see this becoming a trend. Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13907945#comment-13907945 ] stack commented on HDFS-5274: - [~iwasakims] You know, I was thinking... Maybe it ok that there are so many spans? Tracing doesn't cost unless enabled. When debugging, you might want to see in the trace that HDFS is doing a bunch of small reads? Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, HDFS-5274-7.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13908033#comment-13908033
]
Masatake Iwasaki commented on HDFS-5274:
bq. Masatake Iwasaki You know, I was thinking... Maybe it ok that there are so
many spans? Tracing doesn't cost unless enabled. When debugging, you might want
to see in the trace that HDFS is doing a bunch of small reads?
I just missed your comment on uploading v.8 patch.
Because there were so many spans than I expected and receiver's queue was
filled, I think disabling those spans is safer as a starting point.
{noformat}
14/02/19 22:25:23 ERROR impl.ZipkinSpanReceiver: Error trying to append span
(DFSOutputStream.write) to the queue. Blocking Queue was full.
{noformat}
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch,
HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch,
HDFS-5274-7.patch, HDFS-5274-8.patch, Zipkin Trace a06e941b0172ec73.png,
Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13908054#comment-13908054
]
Hadoop QA commented on HDFS-5274:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12630259/ss-5274v8-get.png
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6204//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch,
HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch,
HDFS-5274-7.patch, HDFS-5274-8.patch, Zipkin Trace a06e941b0172ec73.png,
Zipkin Trace d0f0d66b8a258a69.png, ss-5274v8-get.png, ss-5274v8-put.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13906064#comment-13906064
]
Hadoop QA commented on HDFS-5274:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629828/HDFS-5274-7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode
org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6174//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6174//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch,
HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch,
HDFS-5274-7.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace
d0f0d66b8a258a69.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13906167#comment-13906167
]
stack commented on HDFS-5274:
-
My guess is that the failures are unrelated. We can rerun the patch or just
wait on next iteration.
Patch looks great to me. Have you tried it outside of the unit tests to make
sure you get sensible looking spans and numbers? Perhaps I can help here?
Fix these in next patch:
+ * This class do nothing If no SpanReceiver is configured .
+ * Trancing information of HTrace, if exists.
Is formatting ok here?
+ if (source != null) {
+proto.setSource(PBHelper.convertDatanodeInfo(source));
+ }
+ send(out, Op.WRITE_BLOCK, proto.build());
+ } finally {
+ if (ts != null) ts.close();
+}
In BlockReceiver, should traceSpan be getting closed?
Is it possible that below throws an exception?
+ scope.getSpan().addKVAnnotation(
+ stream.getBytes(),
+ jas.getCurrentStream().toString().getBytes());
i..e. we can hope out w/o closing the span since the try/finally only happens
later.
This is in JournalSet in a few places.
TraceInfo and RPCTInfo seem to be same datastructure? Should we define it
onetime only and share?'
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch,
HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch,
HDFS-5274-7.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace
d0f0d66b8a258a69.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13902025#comment-13902025 ] Masatake Iwasaki commented on HDFS-5274: Hi [~eclark], Are you working on this issue now? If you do not have enough time, I would like to help fixing and rebasing this patch. Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13902036#comment-13902036
]
Hadoop QA commented on HDFS-5274:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12607415/HDFS-5274-6.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6158//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch,
HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch,
Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13902095#comment-13902095 ] stack commented on HDFS-5274: - [~iwasakims] I'm pretty sure Elliott would be happy if you took it. I can help (not sure if that a hinderance or a help -- smile). Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13902237#comment-13902237 ] Masatake Iwasaki commented on HDFS-5274: Thanks [~stack]! I'm on it. Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843547#comment-13843547
]
stack commented on HDFS-5274:
-
This facility is excellent.
A few notes on the patch.
Long line and formatting seems off here:
{code}
+ if (header.hasTraceInfo() header.getTraceInfo() != null
header.getTraceInfo().hasTraceId()) {
+String traceDescription = rpcRequest.toString();
+
+// If the incoming RPC included tracing info, always continue the trace
+ TraceInfo parentSpan = new
TraceInfo(header.getTraceInfo().getTraceId(),
+ header.getTraceInfo().getParentId());
+ traceSpan = Trace.startSpan(traceDescription, parentSpan).detach();
+}
{code}
This is a javadoc fix?
- * @param buf - SASL wrapped request of one or more RPCs
+ * @param inBuf - SASL wrapped request of one or more RPCs
Are you breaking your pattern of putting trace string building behind a check
that tracing is enabled in the below?
+ TraceScope traceSpan = Trace.startSpan(Call + method.getName() + to
+
+ remoteId, CLIENT_TRACE_SAMPLER);
nit: declare and define all in the one line rather than do initialization in
constructor: +receivers = new HashSetSpanReceiver();
Any reason we take config on construction and in init for SpanReceiverHost?
SpanReceiverHost is on only when trace is enabled, right? If so, say so in
class comment.
Has to be a shutdown hook? ShutdownHookManager.get().addShutdownHook ? This is
fine unless we envision someone having to override it which I suppose should
never happen for an optionally enabled, rare, trace function?
HTraceConfiguration is for testing only? Should be @visiblefortesting only or
a comment at least?
nit: Do Classname.simple() here instead? +TraceScope scope =
Trace.startSpan(DFSInputStream.class.getSimpleName());
Should there be defines for a few of these? DFSInputStream.close seems
fine... only used once DFSInputStream.read?
Is there something off in the formatting here?
-
- private static class Packet {
-private static final long HEART_BEAT_SEQNO = -1L;
-long seqno; // sequencenumber of buffer in block
-final long offsetInBlock; // offset in block
-boolean syncBlock; // this packet forces the current block to disk
-int numChunks; // number of chunks currently in packet
-final int maxChunks; // max chunks in packet
+ private Span traceSpan = null;
+
+private static class Packet {
+ private static final long HEART_BEAT_SEQNO = -1L;
+ long seqno; // sequencenumber of buffer in block
+ final long offsetInBlock; // offset in block
+ boolean syncBlock; // this packet forces the current block to disk
+ int numChunks; // number of chunks currently in packet
+ final int maxChunks; // max chunks in packet
Or is it just git cleaning whitespace?
Gratuitous change?
-ClientOperationHeaderProto header =
+ClientOperationHeaderProto.Builder builder =
ClientOperationHeaderProto.newBuilder()
.setBaseHeader(buildBaseHeader(blk, blockToken))
-.setClientName(client)
-.build();
-return header;
+.setClientName(client);
+return builder.build();
Is an annotation of 'J' only intentional?
+if (traceSpan != null) {
+ traceSpan.addTimelineAnnotation(J);
Put inside a isTracing check? +Trace.addTimelineAnnotation(Connecting
to downstream + mirrorNode);
ditto: +Trace.addTimelineAnnotation(Waiting for connect ack from
downstream);
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch,
HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch,
Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13833293#comment-13833293 ] Elliott Clark commented on HDFS-5274: - Ping ? Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13789773#comment-13789773
]
Hadoop QA commented on HDFS-5274:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12607415/HDFS-5274-6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5143//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5143//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch,
HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch,
Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13788785#comment-13788785
]
Hadoop QA commented on HDFS-5274:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12607280/HDFS-5274-4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5131//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch,
HDFS-5274-3.patch, HDFS-5274-4.patch, Zipkin Trace a06e941b0172ec73.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13788813#comment-13788813
]
Hadoop QA commented on HDFS-5274:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12607290/HDFS-5274-5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5133//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch,
HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, Zipkin Trace
a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782731#comment-13782731
]
Hadoop QA commented on HDFS-5274:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12606051/HDFS-5274-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestEditLogJournalFailures
org.apache.hadoop.hdfs.qjournal.TestNNWithQJM
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5072//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5072//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch,
Zipkin Trace a06e941b0172ec73.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783445#comment-13783445
]
Hadoop QA commented on HDFS-5274:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12606168/HDFS-5274-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5078//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5078//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch,
HDFS-5274-3.patch, Zipkin Trace a06e941b0172ec73.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
