[jira] [Commented] (HBASE-7879) JUnit dependency in main from htrace
[ https://issues.apache.org/jira/browse/HBASE-7879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13584537#comment-13584537 ] Jonathan Leavitt commented on HBASE-7879: - I did a release of htrace to 1.50 that just has the dependency scope change for junit and no other changes. JUnit dependency in main from htrace Key: HBASE-7879 URL: https://issues.apache.org/jira/browse/HBASE-7879 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.96.0 Reporter: nkeywal Priority: Minor HTrace main depends on Junit , it should be only test. I created a junit in the github, that's https://github.com/cloudera/htrace/issues/1. If it's not fixed, we will be able to drop it in our pom, but let's wait a little before. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13459785#comment-13459785 ] Jonathan Leavitt commented on HBASE-6524: - bq. Doc is great. (it would not be very difficult to remove this requirement). What would this take? Well to remove the client addendum we would have to instrument the entry points into hbase. I will try to find one good class to instrument as the entry point in to the system. HTable.java might work, but it doesn't include all operations, and importantly does not include all of the steps of a 'get', bx getting the HTable is a big part, right? Maybe HTablePool.java would work. bq. Can we do this for hdfs too yet? Not yet. I think Matt or Todd might be working on it. I have been looking into, but it does require all of the additional instrumentation to HDFS. bq. What about an example that enables trace over hdfs too while the Get is going on? I'm not sure what you mean by this. bq. On doc in general, I came across this recent quote Documentation is like sex: when it is good, it is very very good; and when it is bad, it is better than nothing. Great quote. Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Assignee: Jonathan Leavitt Fix For: 0.96.0 Attachments: 6524.addendum, 6524-v2.txt, 6524v3.txt, createTableTrace.png, hbase-6524.diff Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13460247#comment-13460247 ] Jonathan Leavitt commented on HBASE-6524: - Sounds good. :) Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Assignee: Jonathan Leavitt Fix For: 0.96.0 Attachments: 6524.addendum, 6524-v2.txt, 6524v3.txt, createTableTrace.png, hbase-6524.diff Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13456492#comment-13456492 ] Jonathan Leavitt commented on HBASE-6524: - @stack: [here|https://docs.google.com/document/d/11kM7dHetY5LYaC8rIkXfCHoxcbliTb9Xa3SZUfFhDu8/edit] is a link to my ref guide tracing stuff so far. The link is public and anyone can comment on it. Let me know how it looks. Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Assignee: Jonathan Leavitt Fix For: 0.96.0 Attachments: 6524.addendum, 6524-v2.txt, 6524v3.txt, createTableTrace.png, hbase-6524.diff Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13450284#comment-13450284 ] Jonathan Leavitt commented on HBASE-6524: - @stack I agree, I will write something up for the refguide, and I'm also working on a more in-depth blog post. Maybe have them done in a week or two (hopefully less depending on my motivation this weekend). Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Assignee: Jonathan Leavitt Fix For: 0.96.0 Attachments: 6524.addendum, 6524-v2.txt, 6524v3.txt, createTableTrace.png, hbase-6524.diff Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[
https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440631#comment-13440631
]
Jonathan Leavitt commented on HBASE-6524:
-
I'm running the test suite with hadoop 2.0 profile
{code}mvn clean -Dhadoop.profile=2.0 -P localTests test {code}
with and without patch. So far, the same test failures are affecting both.
Will update again once the whole suite is done running.
Hooks for hbase tracing
---
Key: HBASE-6524
URL: https://issues.apache.org/jira/browse/HBASE-6524
Project: HBase
Issue Type: Sub-task
Reporter: Jonathan Leavitt
Assignee: Jonathan Leavitt
Fix For: 0.96.0
Attachments: 6524.addendum, createTableTrace.png, hbase-6524.diff
Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace]
library to add dapper-like tracing to hbase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[
https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440664#comment-13440664
]
Jonathan Leavitt commented on HBASE-6524:
-
Update after running
{code}mvn clean -Dhadoop.profile=2.0 -P localTests test{code}
with and without the patch. Many of the failed tests fail on both, and there
are some failures unique to each. I would bet the results would be different if
I ran the tests again. I believe that some of the tests are not completely
stable for hadoop 2.0.
Hooks for hbase tracing
---
Key: HBASE-6524
URL: https://issues.apache.org/jira/browse/HBASE-6524
Project: HBase
Issue Type: Sub-task
Reporter: Jonathan Leavitt
Assignee: Jonathan Leavitt
Fix For: 0.96.0
Attachments: 6524.addendum, createTableTrace.png, hbase-6524.diff
Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace]
library to add dapper-like tracing to hbase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[
https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440693#comment-13440693
]
Jonathan Leavitt commented on HBASE-6524:
-
I may have some other problem then because here are the failed tests without
htrace:
{code}
Failed tests:
testBulkOutputWithoutAnExistingTable(org.apache.hadoop.hbase.mapreduce.TestImportTsv)
testExcludeMinorCompaction(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat)
testMRIncrementalLoadWithSplit(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat)
testMROnTable(org.apache.hadoop.hbase.mapreduce.TestImportTsv)
testMROnTableWithCustomMapper(org.apache.hadoop.hbase.mapreduce.TestImportTsv)
testMultiRegionTable(org.apache.hadoop.hbase.mapreduce.TestTableMapReduce)
testMultipleTables(org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient):
Archived HFiles
(hdfs://localhost:56525/user/jonathan/hbase/.archive/otherTable/188a593a472f6acc10b09c919b9b0659/fam)
should have gotten deleted, but didn't, remaining files:[hdfs
testMultithreadedTableMapper(org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper)
testRowCounterExclusiveColumn(org.apache.hadoop.hbase.mapreduce.TestRowCounter)
testRowCounterHiddenColumn(org.apache.hadoop.hbase.mapreduce.TestRowCounter)
testRowCounterNoColumn(org.apache.hadoop.hbase.mapreduce.TestRowCounter)
testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat)
Tests in error:
testDisableInactivePeer(org.apache.hadoop.hbase.replication.TestReplication):
Shutting down
testGetRowVersions(org.apache.hadoop.hbase.TestMultiVersions): Shutting down
testMROnTableWithTimestamp(org.apache.hadoop.hbase.mapreduce.TestImportTsv)
testMetaExport(org.apache.hadoop.hbase.mapreduce.TestImportExport)
testMultiRegionTable(org.apache.hadoop.hbase.mapred.TestTableMapReduce): Job
failed!
testScanEmptyToAPP(org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan)
testScanEmptyToEmpty(org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan)
testScanMultipleVersions(org.apache.hadoop.hbase.TestMultiVersions):
org.apache.hadoop.hbase.MasterNotRunningException: Can create a proxy to
master, but it is not running
testScanOPPToEmpty(org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan)
testScanYZYToEmpty(org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan)
testSimpleCase(org.apache.hadoop.hbase.mapreduce.TestImportExport)
testWALPlayer(org.apache.hadoop.hbase.mapreduce.TestWALPlayer)
testWithDeletes(org.apache.hadoop.hbase.mapreduce.TestImportExport)
{code}
Also more than 3.
Hooks for hbase tracing
---
Key: HBASE-6524
URL: https://issues.apache.org/jira/browse/HBASE-6524
Project: HBase
Issue Type: Sub-task
Reporter: Jonathan Leavitt
Assignee: Jonathan Leavitt
Fix For: 0.96.0
Attachments: 6524.addendum, createTableTrace.png, hbase-6524.diff
Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace]
library to add dapper-like tracing to hbase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440754#comment-13440754 ] Jonathan Leavitt commented on HBASE-6524: - I pass TestMultiVersions with the sometimes, and fail it sometimes. The same happens without the patch. Linux jsleavit 3.2.0-27-generic #43-Ubuntu SMP Fri Jul 6 14:25:57 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Assignee: Jonathan Leavitt Fix For: 0.96.0 Attachments: 6524.addendum, 6524-v2.txt, createTableTrace.png, hbase-6524.diff Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[
https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13439723#comment-13439723
]
Jonathan Leavitt commented on HBASE-6524:
-
{quote}There were 90 more failed tests introduced for hadoop 2.0 profile.
I suggest we temporarily rollback the patch and give Jonathan enough time to
iron out the 90+ test failures.{quote}
I think I have the fix. Stay tuned.
Hooks for hbase tracing
---
Key: HBASE-6524
URL: https://issues.apache.org/jira/browse/HBASE-6524
Project: HBase
Issue Type: Sub-task
Reporter: Jonathan Leavitt
Fix For: 0.96.0
Attachments: createTableTrace.png, hbase-6524.diff
Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace]
library to add dapper-like tracing to hbase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13439789#comment-13439789 ] Jonathan Leavitt commented on HBASE-6524: - So I believe the problem was that htrace was depending on hadoop 1.0.3 and there was some problem when running it with hadoop 2 on the classpath. I removed the hadoop dependency from htrace, so now it just uses what it needs, which is better anyways. The only change to the patch is depending on htrace version 1.49 as opposed to 1.48. Should I upload a new patch that just changes the dependency version, or one that has all of the hooks and depends on htrace 1.49? Also, the htrace maven repo won't be synced to maven central for another 2 hours, so I can't upload a patch until then. Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Fix For: 0.96.0 Attachments: createTableTrace.png, hbase-6524.diff Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[
https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13439810#comment-13439810
]
Jonathan Leavitt commented on HBASE-6524:
-
With htrace 1.49 I run:
{code}mvn clean -Dhadoop.profile=2.0 test -Dtest=TestClusterBootOrder{code}
with no problems.
Just to make sure, I'm assuming you got htrace 1.49 snapshot from github? Can
you pull and install again?
Hooks for hbase tracing
---
Key: HBASE-6524
URL: https://issues.apache.org/jira/browse/HBASE-6524
Project: HBase
Issue Type: Sub-task
Reporter: Jonathan Leavitt
Fix For: 0.96.0
Attachments: createTableTrace.png, hbase-6524.diff
Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace]
library to add dapper-like tracing to hbase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13439873#comment-13439873 ] Jonathan Leavitt commented on HBASE-6524: - htrace 1.49 is available in maven now. I've run the whole test suite, and I am still failing some tests when I run with the hadoop 2.0 profile. I will continue to look into it, and put updates here. Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Fix For: 0.96.0 Attachments: createTableTrace.png, hbase-6524.diff Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13439886#comment-13439886 ] Jonathan Leavitt commented on HBASE-6524: - @Zhihong, try running maven with -U flag (forces it to update dependencies). List of failed tests coming soon (have to rerun). Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Fix For: 0.96.0 Attachments: createTableTrace.png, hbase-6524.diff Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[
https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13439924#comment-13439924
]
Jonathan Leavitt commented on HBASE-6524:
-
So I ran the full test suite with:
{code}mvn clean -Dhadoop.profile=2.0 test{code}
which yielded:
{code}
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] HBase . SUCCESS [2.814s]
[INFO] HBase - Common SUCCESS [8.505s]
[INFO] HBase - Hadoop Compatibility .. SUCCESS [1.194s]
[INFO] HBase - Hadoop Two Compatibility .. SUCCESS [3.113s]
[INFO] HBase - Server FAILURE [25:57.982s]
[INFO] HBase - Hadoop One Compatibility .. SKIPPED
[INFO] HBase - Integration Tests . SKIPPED
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 26:14.129s
[INFO] Finished at: Wed Aug 22 15:39:03 PDT 2012
[INFO] Final Memory: 90M/630M
[INFO]
{code}
The failures:
{code}
Failed tests:
testRowCounterNoColumn(org.apache.hadoop.hbase.mapreduce.TestRowCounter)
testRowCounterHiddenColumn(org.apache.hadoop.hbase.mapreduce.TestRowCounter)
testMROnTable(org.apache.hadoop.hbase.mapreduce.TestImportTsv)
testMROnTableWithTimestamp(org.apache.hadoop.hbase.mapreduce.TestImportTsv)
testMROnTableWithCustomMapper(org.apache.hadoop.hbase.mapreduce.TestImportTsv)
testBulkOutputWithoutAnExistingTable(org.apache.hadoop.hbase.mapreduce.TestImportTsv)
Tests in error:
testRowCounterExclusiveColumn(org.apache.hadoop.hbase.mapreduce.TestRowCounter)
testSimpleCase(org.apache.hadoop.hbase.mapreduce.TestImportExport)
testWithDeletes(org.apache.hadoop.hbase.mapreduce.TestImportExport)
{code}
So it seems I fixed most of the problems, but not all of them. will continue to
investigate
Hooks for hbase tracing
---
Key: HBASE-6524
URL: https://issues.apache.org/jira/browse/HBASE-6524
Project: HBase
Issue Type: Sub-task
Reporter: Jonathan Leavitt
Fix For: 0.96.0
Attachments: createTableTrace.png, hbase-6524.diff
Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace]
library to add dapper-like tracing to hbase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13439991#comment-13439991 ] Jonathan Leavitt commented on HBASE-6524: - Strange, I pass some of the tests that you fail (all run locally though). I'm still not sure exactly what's causing these errors, but it seems to me that it must have something to do with some missing dependencies or something, especially considering that tracing is never on for any of the failing tests, meaning very little htrace code is actually getting called run. I did remove the commons-logging dependency from htrace, to no avail. It seems bizarre that using hadoop 2 would cause the tests to fail. I'm fine with removing with the patch so I have more time to figure out exactly what's going on. Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Fix For: 0.96.0 Attachments: 6524.addendum, createTableTrace.png, hbase-6524.diff Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Leavitt updated HBASE-6524: Attachment: createTableTrace.png A trace of a createTable. Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Attachments: createTableTrace.png, hbase-6524.diff Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[
https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13438980#comment-13438980
]
Jonathan Leavitt commented on HBASE-6524:
-
@Stack. I uploaded another image. As you may notice, the descriptions on this
image are not as descriptive as they are in the first image I uploaded. This
is because the toString() on HBaseServer.Call is different with the
ProtobufRpcEngine. It now just concatenates the toString of the rpcRequestBody
with the toString of the connection. Unfortunately the default protobuf
toString does not give the most useful information. You can still make out the
basics of the requests.
bq. Yeah, that sounds good. Can do that in new JIRA?
Yes. I think that this commit is good for including htrace in maven
dependencies and the basic instrumentation, and additional instrumentation
should go in other JIRAs.
{quote}I just think this a topic that will be of general interest.
For refguide, a sentence or two w/ pointers on how to set it all up would be
great. {quote}
I'll try to have some sort of blog post and ref guide stuff done today or
tomorrow.
bq. I think we should commit this if its fine by you.
Sounds good.
Hooks for hbase tracing
---
Key: HBASE-6524
URL: https://issues.apache.org/jira/browse/HBASE-6524
Project: HBase
Issue Type: Sub-task
Reporter: Jonathan Leavitt
Attachments: createTableTrace.png, hbase-6524.diff
Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace]
library to add dapper-like tracing to hbase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13439211#comment-13439211 ] Jonathan Leavitt commented on HBASE-6524: - Sorry about this. I am not sure what is going wrong with hadoop 2.0. I am looking into it. Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Fix For: 0.96.0 Attachments: createTableTrace.png, hbase-6524.diff Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437990#comment-13437990 ] Jonathan Leavitt commented on HBASE-6524: - Final update on findbugs: I installed the findbugs eclipse plugin and inspected all of the files I edited. I saw no bugs introduced by any code from my patch. Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Attachments: htracehooks1.diff Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Leavitt updated HBASE-6524: Attachment: (was: htracehooks1.diff) Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Attachments: hbase-6524.diff Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Leavitt updated HBASE-6524: Attachment: hbase-6524.diff updated diff Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Attachments: hbase-6524.diff Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[
https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13438391#comment-13438391
]
Jonathan Leavitt commented on HBASE-6524:
-
@Stack
bq. Is htrace up in mvn central?
Yes
{quote} Why do we create a new instance of TInfo here rather than just use the
one we have (presuming hasTinfo means the request has a TInfo?)?
+ call = new Call(id, param, this, responder, callSize, new TraceInfo(
+ request.getTinfo().getTraceId(), request.getTinfo().getParentId()));{quote}
I'm trying to keep HTrace agnostic of RPC mechanism, so the TraceInfo in the
request is a protobuf TraceInfo (RPCTinfo from Tracing.proto). I just convert
it to the TraceInfo defined in HTrace to decrease dependency on protobufs.
On the other hand, it's not a big deal to store the RPCTinfo in the Call, so if
you think the extra overhead from crating the TraceInfo is bad, I'm happy to
change it.
bq. Should say what a spanReceiverHost is. When I load receivers, what kind of
resources are we allocating?
Do you mean I should explain SpanReceiverHosts in HMaster? Or describe them
more thoroughly in SpanReceiverHost.java? Or somewhere else?
bq. Is that all it takes to add tracing? That is pretty sweet. I'd say this
patch is close to commit. See comments above. You going to do a bit of a blog
w/ fancy tracing pictures? Something we could reference from the refguide? Any
more pretty pictures for us?
This patch is enough to add tracing for RPCs. There are still places in HBase
that need to be instrumented (by that I mean wrapping any thread changes). It
would also be useful to add some more spans for interesting HBaseEvents (maybe
start a new span any time we talk to HDFS).
I will work on a blog post. I started working on a more thorough README for
[htrace|https://github.com/cloudera/htrace]. After I am happy with that I can
write up something more specific for the refguide if necessary.
Hooks for hbase tracing
---
Key: HBASE-6524
URL: https://issues.apache.org/jira/browse/HBASE-6524
Project: HBase
Issue Type: Sub-task
Reporter: Jonathan Leavitt
Attachments: hbase-6524.diff
Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace]
library to add dapper-like tracing to hbase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436104#comment-13436104 ] Jonathan Leavitt commented on HBASE-6524: - Apologies, I realized that the generated protubufs should not contribute additional findbugs warnings because they should be in findbugx-exclude.xml. I believe I remember hsieh mentioning that changing the pbufs could affect findbug warnings, but maybe I misunderstood. I did however look through all the files I changed at the findbugs warnings for those files and I was unable to find anything. I will keep looking. Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Attachments: htracehooks1.diff Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435664#comment-13435664 ] Jonathan Leavitt commented on HBASE-6524: - In response to hadoop QA: TestReplication passes for me, so I don't think I caused the failure. The new findbugs come from the protobuf generated classes. As for the additional javac compiler warning, I am not completely sure. When I compile myself I only get 4 warnings, not 5. The hadoop QA console output seems to suggest it is a maven problem. I will look into this further. @stack: given the above, what would you recommend I do to make progress on getting this committed? Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Attachments: htracehooks1.diff Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Leavitt updated HBASE-6524: Attachment: htracehooks1.diff added some tests Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Attachments: htracehooks1.diff Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Leavitt updated HBASE-6524: Attachment: (was: patch6.txt) Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Attachments: htracehooks1.diff Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Leavitt updated HBASE-6524: Attachment: (was: patch4.txt) Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Attachments: patch6.txt Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Leavitt updated HBASE-6524: Status: Patch Available (was: Open) uploading updated patch with changes from stack and todd Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Attachments: patch6.txt Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Leavitt updated HBASE-6524: Attachment: patch6.txt new patch with changes from stack and todd. Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Attachments: patch6.txt Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6524) Hooks for hbase tracing
Jonathan Leavitt created HBASE-6524: --- Summary: Hooks for hbase tracing Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Attachments: patch4.txt Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Leavitt updated HBASE-6524: Attachment: patch4.txt First patch for tracing hooks in hbase. Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Attachments: patch4.txt Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13430681#comment-13430681 ] Jonathan Leavitt commented on HBASE-6449: - Just made a subissue of this one that has a diff with the hooks needed for tracing rpcs. The diff is also on [review board|https://reviews.apache.org/r/6454/]. Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Affects Versions: 0.96.0 Reporter: Jonathan Leavitt Labels: tracing Attachments: htrace1.diff, htrace2.diff, trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13430682#comment-13430682 ] Jonathan Leavitt commented on HBASE-6524: - Here's the review board link for the patch I uploaded: [review board|https://reviews.apache.org/r/6454/]. The library is on [github|http://www.github.com/cloudera/htrace] and is in maven central. Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Attachments: patch4.txt Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13427481#comment-13427481 ] Jonathan Leavitt commented on HBASE-6449: - Just a progress update: The tracing library is up on [github | https://github.com/cloudera/htrace]. I am close to having it in maven central, which will make it easy to add to hbase. There are some big API changes coming soon (today?) and once those are done I will try to do the first maven release. I am preparing an hbase patch with the hooks I have now that I will hopefully upload for review early next week. Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Affects Versions: 0.96.0 Reporter: Jonathan Leavitt Labels: tracing Attachments: htrace1.diff, htrace2.diff, trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Leavitt updated HBASE-6449: Attachment: htrace2.diff Second version of the patch. Also available on the review board. Reworking the API today. Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Affects Versions: 0.96.0 Reporter: Jonathan Leavitt Labels: tracing Attachments: htrace1.diff, htrace2.diff, trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423426#comment-13423426 ] Jonathan Leavitt commented on HBASE-6449: - @zhihong Do you think it is best to add it as a module as opposed to keeping it completely independent, and just having hbase depend on it? Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Affects Versions: 0.96.0 Reporter: Jonathan Leavitt Labels: tracing Attachments: htrace1.diff, trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423447#comment-13423447 ] Jonathan Leavitt commented on HBASE-6449: - Sounds good. I will submit a new patch today that deals with most of your and Todd's changes as well as making htrace a module of hbase. Let's just be careful going forward to ensure that the tracing library can easily be broken out when necessary. Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Affects Versions: 0.96.0 Reporter: Jonathan Leavitt Labels: tracing Attachments: htrace1.diff, trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423482#comment-13423482 ] Jonathan Leavitt commented on HBASE-6449: - bq. What do you mean by having hbase dependent on htrace? I meant having htrace as an independent maven project in the hbase source tree and add a dependency for htrace in hbase's pom. The original plan was to get the tracing library committed to hbase first so the hbase committers could review and would be more likely to commit the necessary hooks into hbase. It would also save the initial hassle of getting the library into maven. But you would recommend putting it on github, and only submitting patches with the instrumentation for hbase? If htrace were on github and the hooks were in hbase, would those downloading hbase source just have to grab htrace off github and have maven install it locally? Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Affects Versions: 0.96.0 Reporter: Jonathan Leavitt Labels: tracing Attachments: htrace1.diff, trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423506#comment-13423506 ] Jonathan Leavitt commented on HBASE-6449: - I updated the diff on [review board |https://reviews.apache.org/r/6127/] with many of Todd and Ted's suggestions. Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Affects Versions: 0.96.0 Reporter: Jonathan Leavitt Labels: tracing Attachments: htrace1.diff, trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423524#comment-13423524 ] Jonathan Leavitt commented on HBASE-6449: - @stack you nailed it. The original goal was just tracing for hbase, but it seemed silly to not put in the extra effort necessary to keep the tracing code generic enough for use elsewhere. So it seems the consensus is that I should get the tracing code up on github and develop it there? Then I would submit patches that just integrate the library into htrace? Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Affects Versions: 0.96.0 Reporter: Jonathan Leavitt Labels: tracing Attachments: htrace1.diff, trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6449) Dapper like tracing
Jonathan Leavitt created HBASE-6449: --- Summary: Dapper like tracing Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Reporter: Jonathan Leavitt Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13421650#comment-13421650 ] Jonathan Leavitt commented on HBASE-6449: - [Dapper | http://research.google.com/pubs/pub36356.html] is google's system to trace requests through their systems. The paper discusses how useful Google engineers find the system, as well as laying out the basic design of the system. Accumulo added tracing to their project based on the Dapper design. They call it cloudtrace. I have used Accumulo's cloudtrace as a starting point for a tracing library. I modified it to not be dependent on any particular RPC framework, as well as adding more detailed sampling, among other changes. I plan to commit this to the hbase source tree as an independent maven project. I will then submit commits that properly instrument HBase to use the tracing library. This requires adding an optional payload to the protobuf RPC's containing two 64bit id's: one for the trace id and another for the unique id of the span that sent the message. There will be no additional data in the messages when tracing is not active. The tracing library works by storing information in ThreadLocal's. In order to preserve this information across thread changes, the thread changes must be tracked. The tracing package accomplishes this by wrapping any thread changes to pass the information through to the new thread. I have done most of the instrumentation for HBase. Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Reporter: Jonathan Leavitt Labels: tracing Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Leavitt updated HBASE-6449: Attachment: trace.png This is just a sample trace that I made with my instrumented version of hbase. It shows the operations involved in a delete table operation. To create this, I had the tracing library write the spans to local files, which I piped into a simple python script to make a graphviz graph. Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Reporter: Jonathan Leavitt Labels: tracing Attachments: trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13421806#comment-13421806 ] Jonathan Leavitt commented on HBASE-6449: - @Zhihong I was planning to keep the library as independent as possible so it can be broken out easily later and used for other hadoop systems. What are your thoughts on this? Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Reporter: Jonathan Leavitt Labels: tracing Attachments: trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Leavitt updated HBASE-6449: Affects Version/s: 0.96.0 Release Note: Adds htrace tracing library to hbase source tree. Status: Patch Available (was: Open) This is the initial commit of the htrace library. Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Affects Versions: 0.96.0 Reporter: Jonathan Leavitt Labels: tracing Attachments: htrace1.diff, trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Leavitt updated HBASE-6449: Attachment: htrace1.diff Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Affects Versions: 0.96.0 Reporter: Jonathan Leavitt Labels: tracing Attachments: htrace1.diff, trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13421927#comment-13421927 ] Jonathan Leavitt commented on HBASE-6449: - [Link to review board for htrace1.diff.| https://reviews.apache.org/r/6127/] Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Affects Versions: 0.96.0 Reporter: Jonathan Leavitt Labels: tracing Attachments: htrace1.diff, trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
