[jira] [Commented] (HBASE-4510) HDFS-1620 related changes downstream (For compiling with HDFS 0.23+)
[ https://issues.apache.org/jira/browse/HBASE-4510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117025#comment-13117025 ] Ted Yu commented on HBASE-4510: --- I think the above diff is in the right direction. HDFS-1620 related changes downstream (For compiling with HDFS 0.23+) Key: HBASE-4510 URL: https://issues.apache.org/jira/browse/HBASE-4510 Project: HBase Issue Type: Task Affects Versions: 0.94.0 Reporter: Harsh J Assignee: Harsh J Priority: Blocker HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out. Two solutions: * We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?) * HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog
[ https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HBASE-4487: Attachment: appendNoSync4.txt The increment operation releases the rowlock before doing the sync to the HLog. This improves performance of increments on hot rows. Introuced method HLog.appendNoSync() that returns a txid. The increment method then release the rowlock and invokes HLog.sync(txid). The HLog.sync(txid) returns only if all the transactions upto the one identified by that txid has been successfully sycned to HDFS. The increment operation can release the rowlock before sync-ing the Hlog Key: HBASE-4487 URL: https://issues.apache.org/jira/browse/HBASE-4487 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: appendNoSync4.txt This allows for better throughput when there are hot rows.I have seen this change make a single row update improve from 400 increments/sec/server to 4000 increments/sec/server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4503) Purge deprecated HBaseClusterTestCase
[ https://issues.apache.org/jira/browse/HBASE-4503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4503: - Attachment: 4503-v2.txt This should do it. Removes HBaseClusterTestCase and the MultiTable thingy used by MR. Groups a few of old dependent tests and leaves other alone. Should save us a few minutes. Only 120+ to go. Purge deprecated HBaseClusterTestCase - Key: HBASE-4503 URL: https://issues.apache.org/jira/browse/HBASE-4503 Project: HBase Issue Type: Improvement Reporter: stack Attachments: 4503-v2.txt, 4503.txt It could gain us a few minutes on overall test run in the cases where we don't spin up a cluster for each test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4503) Purge deprecated HBaseClusterTestCase
[ https://issues.apache.org/jira/browse/HBASE-4503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack reassigned HBASE-4503: Assignee: stack Purge deprecated HBaseClusterTestCase - Key: HBASE-4503 URL: https://issues.apache.org/jira/browse/HBASE-4503 Project: HBase Issue Type: Improvement Reporter: stack Assignee: stack Attachments: 4503-v2.txt, 4503.txt It could gain us a few minutes on overall test run in the cases where we don't spin up a cluster for each test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4503) Purge deprecated HBaseClusterTestCase
[ https://issues.apache.org/jira/browse/HBASE-4503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117036#comment-13117036 ] stack commented on HBASE-4503: -- Will post to RB in morn after I give it another run though. Purge deprecated HBaseClusterTestCase - Key: HBASE-4503 URL: https://issues.apache.org/jira/browse/HBASE-4503 Project: HBase Issue Type: Improvement Reporter: stack Assignee: stack Attachments: 4503-v2.txt, 4503.txt It could gain us a few minutes on overall test run in the cases where we don't spin up a cluster for each test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4494) AvroServer:: get fails with NPE on a non-existent row
[ https://issues.apache.org/jira/browse/HBASE-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117038#comment-13117038 ] Kay Kay commented on HBASE-4494: yes - i am :) AvroServer:: get fails with NPE on a non-existent row - Key: HBASE-4494 URL: https://issues.apache.org/jira/browse/HBASE-4494 Project: HBase Issue Type: Bug Components: avro Affects Versions: 0.90.4 Reporter: Kay Kay Assignee: Kay Kay Fix For: 0.90.5 Attachments: HBASE-4494.patch Try to submit a get request to the avro gateway. If the row specified for a given table does not exist, the server request fails with a NPE. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog
[
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117047#comment-13117047
]
Ted Yu commented on HBASE-4487:
---
Nice feature.
{code}
+ System.err.println(Version Mismatch between client and server +
+ ... command aborted.);
{code}
I think LOG should be used.
Incrementer thread should be set as Daemon thread.
For HLog.unflushedEntries, since it is used to generate txid, I wonder if there
is a better name for it.
{code}
+// method is pretty heavyweight as far a locking is concerned. The
{code}
Should be as far as ...
{code}
+ // write out all accumulated Entries to hdfs.
+ for (Entry e : pending) {
+writer.append(e);
+ }
{code}
I think exception handling should be added here. What if writer.append() raises
exception ? The remaining Entries would be lost ?
The increment operation can release the rowlock before sync-ing the Hlog
Key: HBASE-4487
URL: https://issues.apache.org/jira/browse/HBASE-4487
Project: HBase
Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
Attachments: appendNoSync4.txt
This allows for better throughput when there are hot rows.I have seen this
change make a single row update improve from 400 increments/sec/server to
4000 increments/sec/server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog
[
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dhruba borthakur updated HBASE-4487:
Attachment: appendNoSync5.txt
Addressed Ted Yu's review comments. The code that does
{code}
for (Entry e : pending) {
+writer.append(e);
+ }
{code}
does not catch exceptions, instead throws an exception to the caller if any of
the edits fail to make it to HDFS. In fact, Hbase regionserver exits if an HDFS
write/sync fails, this is expected behaviour.
The increment operation can release the rowlock before sync-ing the Hlog
Key: HBASE-4487
URL: https://issues.apache.org/jira/browse/HBASE-4487
Project: HBase
Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
Attachments: appendNoSync4.txt, appendNoSync5.txt
This allows for better throughput when there are hot rows.I have seen this
change make a single row update improve from 400 increments/sec/server to
4000 increments/sec/server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog
[ https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117074#comment-13117074 ] Ted Yu commented on HBASE-4487: --- Wow, this is the quickest turnaround for code review I have ever seen :-) Normally you can wait for other people's comments before making the next patch. I still see System.err.println() call. For my last comment, thanks for reminding me the current behavior. What I meant was that your change introduced some kind of buffering which would delay the bubbling of IOException. I guess this is Okay. We should document this in release notes. Can I ask my favorite question ? Test suite. Also, using https://reviews.apache.org/ would be convenient. The increment operation can release the rowlock before sync-ing the Hlog Key: HBASE-4487 URL: https://issues.apache.org/jira/browse/HBASE-4487 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: appendNoSync4.txt, appendNoSync5.txt This allows for better throughput when there are hot rows.I have seen this change make a single row update improve from 400 increments/sec/server to 4000 increments/sec/server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4509) [hbck] Improve region map output
[ https://issues.apache.org/jira/browse/HBASE-4509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117093#comment-13117093 ] jirapos...@reviews.apache.org commented on HBASE-4509: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2111/ --- Review request for hbase, Todd Lipcon and Michael Stack. Summary --- This is on top of HBASE-4506 but doesn't require it. commit 17d3b8856a8e7c803463fb37b1bd4691d0a02ba9 Author: Jonathan Hsieh j...@cloudera.com Date: Wed Sep 28 20:32:02 2011 -0700 HBASE-4509 [hbck] Improve region map output Details: HBASE-4375 added a region coverage visualization to hbck in details mode. When users have binary row keys the output is difficult to parse (awk/sed) or pull into programs (numeric, excel) capable of handling tsv formatted data. This patch: * improves output by using Bytes.toStringBinary (which escapes binary) instead of Bytes.toString when printing keys, * suggests some repair actions, and * collects overlap group that groups regions that are overlapping. This addresses bug HBASE-4509. https://issues.apache.org/jira/browse/HBASE-4509 Diffs - src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 8465724 src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java fae0881 Diff: https://reviews.apache.org/r/2111/diff Testing --- Updated unit test passes. Used on a live cluster to get better information when dianosing problem. Thanks, jmhsieh [hbck] Improve region map output Key: HBASE-4509 URL: https://issues.apache.org/jira/browse/HBASE-4509 Project: HBase Issue Type: Improvement Affects Versions: 0.92.0, 0.94.0, 0.90.5 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: 0001-HBASE-4509-hbck-Improve-region-map-output.patch HBASE-4375 added a region coverage visualization to hbck in details mode. When users have binary row keys the output is difficult to parse (awk/sed) or pull into programs (numeric, excel) capable of handling tsv formatted data. This patch * improves output by using Bytes.toStringBinary (which escapes binary) instead of Bytes.toString when printing keys, * suggests some repair actions, and * collects problem group that groups regions that are overlapping. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4377) [hbck] Offline rebuild .META. from fs data only.
[
https://issues.apache.org/jira/browse/HBASE-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117095#comment-13117095
]
Jonathan Hsieh commented on HBASE-4377:
---
I'm having a hard time with tests that restart the test hbase mini cluster. I
start cluster, modify meta/hdfs regions, shutdown cluster, rebuild meta, and
then get an NPE when restarting.
Specifically, this method sometimes returns null which later causes an NPE when
constructor calls
{code}
User.HadoopUser.init
ugi = (UserGroupInformation) callStatic(getCurrentUGI);
{code}
Test were passing at one point but I can't seem to figure out a direct cause
for why this would fail. Any hints?
[hbck] Offline rebuild .META. from fs data only.
Key: HBASE-4377
URL: https://issues.apache.org/jira/browse/HBASE-4377
Project: HBase
Issue Type: New Feature
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
In a worst case situation, it may be helpful to have an offline .META.
rebuilder that just looks at the file system's .regioninfos and rebuilds meta
from scratch. Users could move bad regions out until there is a clean
rebuild.
It would likely fill in region split holes. Follow on work could given
options to merge or select regions that overlap, or do online rebuilds.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4477) Ability for an application to store metadata into the transaction log
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HBASE-4477: Attachment: coprocessorPut1.txt Implemented andrew's suggestion of enahncing the prePut, postPut, preDelete and postDelete apis to take in the Put/Delete object itself. In the process of running tests. Ability for an application to store metadata into the transaction log - Key: HBASE-4477 URL: https://issues.apache.org/jira/browse/HBASE-4477 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: coprocessorPut1.txt, hlogMetadata1.txt mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE. The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4511) There is data loss when master failovers
There is data loss when master failovers Key: HBASE-4511 URL: https://issues.apache.org/jira/browse/HBASE-4511 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: gaojinchao Priority: Critical Fix For: 0.92.0 It goes like this: Master crashed , at the same time RS with meta is crashing, but RS doesn't eixt. Master startups again and finds all living RS. Master verifies the meta failed, because this RS is crashing. Master reassigns the meta, but it doesn't split the Hlog. So some meta data is loss. About the logs of a failover test case fail. //It said that we want to kill a RS 2011-09-28 19:54:45,694 INFO [Thread-988] regionserver.HRegionServer(1443): STOPPED: Killing for unit test 2011-09-28 19:54:45,694 INFO [Thread-988] master.TestMasterFailover(1007): RS 192.168.2.102,54385,1317264874629 killed //Rs didn't crash. 2011-09-28 19:54:51,763 INFO [Master:0;192.168.2.102,54557,1317264885720] master.HMaster(458): Registering server found up in zk: 192.168.2.102,54385,1317264874629 2011-09-28 19:54:51,763 INFO [Master:0;192.168.2.102,54557,1317264885720] master.ServerManager(232): Registering server=192.168.2.102,54385,1317264874629 2011-09-28 19:54:51,770 DEBUG [Master:0;192.168.2.102,54557,1317264885720] zookeeper.ZKUtil(491): master:54557-0x132b31adbb30005 Unable to get data of znode /hbase/unassigned/1028785192 because node does not exist (not an error) 2011-09-28 19:54:51,771 DEBUG [Master:0;192.168.2.102,54557,1317264885720] zookeeper.ZKUtil(1003): master:54557-0x132b31adbb30005 Retrieved 33 byte(s) of data from znode /hbase/root-region-server and set watcher; 192.168.2.102,54383,131726487... //Meta verification failed and ressigned the meta. So all the regions in the meta is loss. 2011-09-28 19:54:51,773 INFO [Master:0;192.168.2.102,54557,1317264885720] catalog.CatalogTracker(476): Failed verification of .META.,,1 at address=192.168.2.102,54385,1317264874629; org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server 192.168.2.102,54385,1317264874629 not running, aborting 2011-09-28 19:54:51,773 DEBUG [Master:0;192.168.2.102,54557,1317264885720] catalog.CatalogTracker(316): new .META. server: 192.168.2.102,54385,1317264874629 isn't valid. Cached .META. server: null 2011-09-28 19:54:52,274 DEBUG [Master:0;192.168.2.102,54557,1317264885720] zookeeper.ZKUtil(1003): master:54557-0x132b31adbb30005 Retrieved 33 byte(s) of data from znode /hbase/root-region-server and set watcher; 192.168.2.102,54383,131726487... 2011-09-28 19:54:52,277 INFO [Master:0;192.168.2.102,54557,1317264885720] catalog.CatalogTracker(476): Failed verification of .META.,,1 at address=192.168.2.102,54385,1317264874629; org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server 192.168.2.102,54385,1317264874629 not running, aborting 2011-09-28 19:54:52,277 DEBUG [Master:0;192.168.2.102,54557,1317264885720] catalog.CatalogTracker(316): new .META. server: 192.168.2.102,54385,1317264874629 isn't valid. Cached .META. server: null 2011-09-28 19:54:52,778 DEBUG [Master:0;192.168.2.102,54557,1317264885720] zookeeper.ZKUtil(1003): master:54557-0x132b31adbb30005 Retrieved 33 byte(s) of data from znode /hbase/root-region-server and set watcher; 192.168.2.102,54383,131726487... 2011-09-28 19:54:52,782 INFO [Master:0;192.168.2.102,54557,1317264885720] catalog.CatalogTracker(476): Failed verification of .META.,,1 at address=192.168.2.102,54385,1317264874629; org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server 192.168.2.102,54385,1317264874629 not running, aborting 2011-09-28 19:54:52,782 DEBUG [Master:0;192.168.2.102,54557,1317264885720] catalog.CatalogTracker(316): new .META. server: 192.168.2.102,54385,1317264874629 isn't valid. Cached .META. server: null 2011-09-28 19:54:52,782 DEBUG [Master:0;192.168.2.102,54557,1317264885720] zookeeper.ZKAssign(264): master:54557-0x132b31adbb30005 Creating (or updating) unassigned node for 1028785192 with OFFLINE state 2011-09-28 19:54:52,825 DEBUG [Thread-988-EventThread] zookeeper.ZooKeeperWatcher(233): master:54557-0x132b31adbb30005 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/unassigned/1028785192 //It said that Master clean the cluster. 2011-09-28 19:54:52,889 INFO [Master:0;192.168.2.102,54557,1317264885720] master.AssignmentManager(383): Clean cluster startup. Assigning userregions 2011-09-28 19:54:52,889 DEBUG [Master:0;192.168.2.102,54557,1317264885720] zookeeper.ZKAssign(494): master:54557-0x132b31adbb30005 Deleting any existing unassigned nodes -- This message is
[jira] [Updated] (HBASE-4511) There is data loss when master failovers
[ https://issues.apache.org/jira/browse/HBASE-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gaojinchao updated HBASE-4511: -- Attachment: org.apache.hadoop.hbase.master.TestMasterFailover-output.rar Some logs are provided by Ted There is data loss when master failovers Key: HBASE-4511 URL: https://issues.apache.org/jira/browse/HBASE-4511 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: gaojinchao Priority: Critical Fix For: 0.92.0 Attachments: org.apache.hadoop.hbase.master.TestMasterFailover-output.rar It goes like this: Master crashed , at the same time RS with meta is crashing, but RS doesn't eixt. Master startups again and finds all living RS. Master verifies the meta failed, because this RS is crashing. Master reassigns the meta, but it doesn't split the Hlog. So some meta data is loss. About the logs of a failover test case fail. //It said that we want to kill a RS 2011-09-28 19:54:45,694 INFO [Thread-988] regionserver.HRegionServer(1443): STOPPED: Killing for unit test 2011-09-28 19:54:45,694 INFO [Thread-988] master.TestMasterFailover(1007): RS 192.168.2.102,54385,1317264874629 killed //Rs didn't crash. 2011-09-28 19:54:51,763 INFO [Master:0;192.168.2.102,54557,1317264885720] master.HMaster(458): Registering server found up in zk: 192.168.2.102,54385,1317264874629 2011-09-28 19:54:51,763 INFO [Master:0;192.168.2.102,54557,1317264885720] master.ServerManager(232): Registering server=192.168.2.102,54385,1317264874629 2011-09-28 19:54:51,770 DEBUG [Master:0;192.168.2.102,54557,1317264885720] zookeeper.ZKUtil(491): master:54557-0x132b31adbb30005 Unable to get data of znode /hbase/unassigned/1028785192 because node does not exist (not an error) 2011-09-28 19:54:51,771 DEBUG [Master:0;192.168.2.102,54557,1317264885720] zookeeper.ZKUtil(1003): master:54557-0x132b31adbb30005 Retrieved 33 byte(s) of data from znode /hbase/root-region-server and set watcher; 192.168.2.102,54383,131726487... //Meta verification failed and ressigned the meta. So all the regions in the meta is loss. 2011-09-28 19:54:51,773 INFO [Master:0;192.168.2.102,54557,1317264885720] catalog.CatalogTracker(476): Failed verification of .META.,,1 at address=192.168.2.102,54385,1317264874629; org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server 192.168.2.102,54385,1317264874629 not running, aborting 2011-09-28 19:54:51,773 DEBUG [Master:0;192.168.2.102,54557,1317264885720] catalog.CatalogTracker(316): new .META. server: 192.168.2.102,54385,1317264874629 isn't valid. Cached .META. server: null 2011-09-28 19:54:52,274 DEBUG [Master:0;192.168.2.102,54557,1317264885720] zookeeper.ZKUtil(1003): master:54557-0x132b31adbb30005 Retrieved 33 byte(s) of data from znode /hbase/root-region-server and set watcher; 192.168.2.102,54383,131726487... 2011-09-28 19:54:52,277 INFO [Master:0;192.168.2.102,54557,1317264885720] catalog.CatalogTracker(476): Failed verification of .META.,,1 at address=192.168.2.102,54385,1317264874629; org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server 192.168.2.102,54385,1317264874629 not running, aborting 2011-09-28 19:54:52,277 DEBUG [Master:0;192.168.2.102,54557,1317264885720] catalog.CatalogTracker(316): new .META. server: 192.168.2.102,54385,1317264874629 isn't valid. Cached .META. server: null 2011-09-28 19:54:52,778 DEBUG [Master:0;192.168.2.102,54557,1317264885720] zookeeper.ZKUtil(1003): master:54557-0x132b31adbb30005 Retrieved 33 byte(s) of data from znode /hbase/root-region-server and set watcher; 192.168.2.102,54383,131726487... 2011-09-28 19:54:52,782 INFO [Master:0;192.168.2.102,54557,1317264885720] catalog.CatalogTracker(476): Failed verification of .META.,,1 at address=192.168.2.102,54385,1317264874629; org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server 192.168.2.102,54385,1317264874629 not running, aborting 2011-09-28 19:54:52,782 DEBUG [Master:0;192.168.2.102,54557,1317264885720] catalog.CatalogTracker(316): new .META. server: 192.168.2.102,54385,1317264874629 isn't valid. Cached .META. server: null 2011-09-28 19:54:52,782 DEBUG [Master:0;192.168.2.102,54557,1317264885720] zookeeper.ZKAssign(264): master:54557-0x132b31adbb30005 Creating (or updating) unassigned node for 1028785192 with OFFLINE state 2011-09-28 19:54:52,825 DEBUG [Thread-988-EventThread] zookeeper.ZooKeeperWatcher(233): master:54557-0x132b31adbb30005 Received ZooKeeper Event,
[jira] [Commented] (HBASE-4212) TestMasterFailover fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117137#comment-13117137
]
gaojinchao commented on HBASE-4212:
---
@Ted.
I have analyzed the problem that you said and opened a new
issue(https://issues.apache.org/jira/browse/HBASE-4511).
It seems that some data is loss. You can check my analysis.
TestMasterFailover fails occasionally
-
Key: HBASE-4212
URL: https://issues.apache.org/jira/browse/HBASE-4212
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.4
Reporter: gaojinchao
Assignee: gaojinchao
Fix For: 0.90.5
Attachments: HBASE-4212_TrunkV1.patch, HBASE-4212_branch90V1.patch
It seems a bug. The root in RIT can't be moved..
In the failover process, it enforces root on-line. But not clean zk node.
test will wait forever.
void processFailover() throws KeeperException, IOException,
InterruptedException {
// we enforce on-line root.
HServerInfo hsi =
this.serverManager.getHServerInfo(this.catalogTracker.getMetaLocation());
regionOnline(HRegionInfo.FIRST_META_REGIONINFO, hsi);
hsi =
this.serverManager.getHServerInfo(this.catalogTracker.getRootLocation());
regionOnline(HRegionInfo.ROOT_REGIONINFO, hsi);
It seems that we should wait finished as meta region
int assignRootAndMeta()
throws InterruptedException, IOException, KeeperException {
int assigned = 0;
long timeout = this.conf.getLong(hbase.catalog.verification.timeout,
1000);
// Work on ROOT region. Is it in zk in transition?
boolean rit = this.assignmentManager.
processRegionInTransitionAndBlockUntilAssigned(HRegionInfo.ROOT_REGIONINFO);
if (!catalogTracker.verifyRootRegionLocation(timeout)) {
this.assignmentManager.assignRoot();
this.catalogTracker.waitForRoot();
//we need add this code and guarantee that the transition has completed
this.assignmentManager.waitForAssignment(HRegionInfo.ROOT_REGIONINFO);
assigned++;
}
logs:
2011-08-16 07:45:40,715 DEBUG
[RegionServer:0;C4S2.site,47710,1313495126115-EventThread]
zookeeper.ZooKeeperWatcher(252): regionserver:47710-0x131d2690f780004
Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,715 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
zookeeper.ZKAssign(712): regionserver:47710-0x131d2690f780004 Successfully
transitioned node 70236052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-08-16 07:45:40,715 DEBUG [Thread-760-EventThread]
zookeeper.ZooKeeperWatcher(252): master:60701-0x131d2690f780009 Received
ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,716 INFO [PostOpenDeployTasks:70236052]
catalog.RootLocationEditor(62): Setting ROOT region location in ZooKeeper as
C4S2.site:47710
2011-08-16 07:45:40,716 DEBUG [Thread-760-EventThread]
zookeeper.ZKUtil(1109): master:60701-0x131d2690f780009 Retrieved 52 byte(s)
of data from znode /hbase/unassigned/70236052 and set watcher;
region=-ROOT-,,0, server=C4S2.site,47710,1313495126115,
state=RS_ZK_REGION_OPENING
2011-08-16 07:45:40,717 DEBUG [Thread-760-EventThread]
master.AssignmentManager(477): Handling transition=RS_ZK_REGION_OPENING,
server=C4S2.site,47710,1313495126115, region=70236052/-ROOT-
2011-08-16 07:45:40,725 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
zookeeper.ZKAssign(661): regionserver:47710-0x131d2690f780004 Attempting to
transition node 70236052/-ROOT- from RS_ZK_REGION_OPENING to
RS_ZK_REGION_OPENED
2011-08-16 07:45:40,727 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
zookeeper.ZKUtil(1109): regionserver:47710-0x131d2690f780004 Retrieved 52
byte(s) of data from znode /hbase/unassigned/70236052; data=region=-ROOT-,,0,
server=C4S2.site,47710,1313495126115, state=RS_ZK_REGION_OPENING
2011-08-16 07:45:40,740 DEBUG
[RegionServer:0;C4S2.site,47710,1313495126115-EventThread]
zookeeper.ZooKeeperWatcher(252): regionserver:47710-0x131d2690f780004
Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,740 DEBUG [Thread-760-EventThread]
zookeeper.ZooKeeperWatcher(252): master:60701-0x131d2690f780009 Received
ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,740 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
zookeeper.ZKAssign(712): regionserver:47710-0x131d2690f780004 Successfully
transitioned node 70236052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
2011-08-16 07:45:40,741 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
[jira] [Commented] (HBASE-4212) TestMasterFailover fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117145#comment-13117145
]
gaojinchao commented on HBASE-4212:
---
Version :0.90
Results :
Tests run: 649, Failures: 0, Errors: 0, Skipped: 1
T E S T S
---
Running org.apache.hadoop.hbase.master.TestMasterFailover
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 99.09 sec
Results :
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
TestMasterFailover fails occasionally
-
Key: HBASE-4212
URL: https://issues.apache.org/jira/browse/HBASE-4212
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.4
Reporter: gaojinchao
Assignee: gaojinchao
Fix For: 0.90.5
Attachments: HBASE-4212_TrunkV1.patch, HBASE-4212_branch90V1.patch
It seems a bug. The root in RIT can't be moved..
In the failover process, it enforces root on-line. But not clean zk node.
test will wait forever.
void processFailover() throws KeeperException, IOException,
InterruptedException {
// we enforce on-line root.
HServerInfo hsi =
this.serverManager.getHServerInfo(this.catalogTracker.getMetaLocation());
regionOnline(HRegionInfo.FIRST_META_REGIONINFO, hsi);
hsi =
this.serverManager.getHServerInfo(this.catalogTracker.getRootLocation());
regionOnline(HRegionInfo.ROOT_REGIONINFO, hsi);
It seems that we should wait finished as meta region
int assignRootAndMeta()
throws InterruptedException, IOException, KeeperException {
int assigned = 0;
long timeout = this.conf.getLong(hbase.catalog.verification.timeout,
1000);
// Work on ROOT region. Is it in zk in transition?
boolean rit = this.assignmentManager.
processRegionInTransitionAndBlockUntilAssigned(HRegionInfo.ROOT_REGIONINFO);
if (!catalogTracker.verifyRootRegionLocation(timeout)) {
this.assignmentManager.assignRoot();
this.catalogTracker.waitForRoot();
//we need add this code and guarantee that the transition has completed
this.assignmentManager.waitForAssignment(HRegionInfo.ROOT_REGIONINFO);
assigned++;
}
logs:
2011-08-16 07:45:40,715 DEBUG
[RegionServer:0;C4S2.site,47710,1313495126115-EventThread]
zookeeper.ZooKeeperWatcher(252): regionserver:47710-0x131d2690f780004
Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,715 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
zookeeper.ZKAssign(712): regionserver:47710-0x131d2690f780004 Successfully
transitioned node 70236052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-08-16 07:45:40,715 DEBUG [Thread-760-EventThread]
zookeeper.ZooKeeperWatcher(252): master:60701-0x131d2690f780009 Received
ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,716 INFO [PostOpenDeployTasks:70236052]
catalog.RootLocationEditor(62): Setting ROOT region location in ZooKeeper as
C4S2.site:47710
2011-08-16 07:45:40,716 DEBUG [Thread-760-EventThread]
zookeeper.ZKUtil(1109): master:60701-0x131d2690f780009 Retrieved 52 byte(s)
of data from znode /hbase/unassigned/70236052 and set watcher;
region=-ROOT-,,0, server=C4S2.site,47710,1313495126115,
state=RS_ZK_REGION_OPENING
2011-08-16 07:45:40,717 DEBUG [Thread-760-EventThread]
master.AssignmentManager(477): Handling transition=RS_ZK_REGION_OPENING,
server=C4S2.site,47710,1313495126115, region=70236052/-ROOT-
2011-08-16 07:45:40,725 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
zookeeper.ZKAssign(661): regionserver:47710-0x131d2690f780004 Attempting to
transition node 70236052/-ROOT- from RS_ZK_REGION_OPENING to
RS_ZK_REGION_OPENED
2011-08-16 07:45:40,727 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
zookeeper.ZKUtil(1109): regionserver:47710-0x131d2690f780004 Retrieved 52
byte(s) of data from znode /hbase/unassigned/70236052; data=region=-ROOT-,,0,
server=C4S2.site,47710,1313495126115, state=RS_ZK_REGION_OPENING
2011-08-16 07:45:40,740 DEBUG
[RegionServer:0;C4S2.site,47710,1313495126115-EventThread]
zookeeper.ZooKeeperWatcher(252): regionserver:47710-0x131d2690f780004
Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,740 DEBUG [Thread-760-EventThread]
zookeeper.ZooKeeperWatcher(252): master:60701-0x131d2690f780009 Received
ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,740 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
zookeeper.ZKAssign(712): regionserver:47710-0x131d2690f780004 Successfully
[jira] [Commented] (HBASE-4212) TestMasterFailover fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117176#comment-13117176
]
ramkrishna.s.vasudevan commented on HBASE-4212:
---
@Ted/@Gao
I think the issue exists in 0.90 branch and not in trunk. Gao's analysis is
valid in 0.90.
In 0.90
{code}
assignRootAndMeta();
// Is this fresh start with no regions assigned or are we a master joining
// an already-running cluster? If regionsCount == 0, then for sure a
// fresh start. TOOD: Be fancier. If regionsCount == 2, perhaps the
// 2 are .META. and -ROOT- and we should fall into the fresh startup
// branch below. For now, do processFailover.
if (regionCount == 0) {
LOG.info(Master startup proceeding: cluster startup);
this.assignmentManager.cleanoutUnassigned();
this.assignmentManager.assignAllUserRegions();
} else {
LOG.info(Master startup proceeding: master failover);
this.assignmentManager.processFailover();
}
{code}
Inside AssignmentManager.processFailOver()
{code}
HServerInfo hsi =
this.serverManager.getHServerInfo(this.catalogTracker.getMetaLocation());
regionOnline(HRegionInfo.FIRST_META_REGIONINFO, hsi);
hsi =
this.serverManager.getHServerInfo(this.catalogTracker.getRootLocation());
regionOnline(HRegionInfo.ROOT_REGIONINFO, hsi);
{code}
we first call regionOnline inside which the region in RIT is getting removed.
So even before the call back happens for OPENED state it is getting removed.
Hence ROOT node remains in RIT and testcase timesout.
In trunk we dont have this problem as in Assignment.joinCluster() we dont have
this code of removing from regionOnline.
{code}
// Scan META to build list of existing regions, servers, and assignment
// Returns servers who have not checked in (assumed dead) and their regions
MapServerName,ListPairHRegionInfo,Result deadServers =
rebuildUserRegions();
{code}
I will now verify this testcase with the patch with the script in HBASE-4480.
Gao and Ted what do you suggest?
TestMasterFailover fails occasionally
-
Key: HBASE-4212
URL: https://issues.apache.org/jira/browse/HBASE-4212
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.4
Reporter: gaojinchao
Assignee: gaojinchao
Fix For: 0.90.5
Attachments: HBASE-4212_TrunkV1.patch, HBASE-4212_branch90V1.patch
It seems a bug. The root in RIT can't be moved..
In the failover process, it enforces root on-line. But not clean zk node.
test will wait forever.
void processFailover() throws KeeperException, IOException,
InterruptedException {
// we enforce on-line root.
HServerInfo hsi =
this.serverManager.getHServerInfo(this.catalogTracker.getMetaLocation());
regionOnline(HRegionInfo.FIRST_META_REGIONINFO, hsi);
hsi =
this.serverManager.getHServerInfo(this.catalogTracker.getRootLocation());
regionOnline(HRegionInfo.ROOT_REGIONINFO, hsi);
It seems that we should wait finished as meta region
int assignRootAndMeta()
throws InterruptedException, IOException, KeeperException {
int assigned = 0;
long timeout = this.conf.getLong(hbase.catalog.verification.timeout,
1000);
// Work on ROOT region. Is it in zk in transition?
boolean rit = this.assignmentManager.
processRegionInTransitionAndBlockUntilAssigned(HRegionInfo.ROOT_REGIONINFO);
if (!catalogTracker.verifyRootRegionLocation(timeout)) {
this.assignmentManager.assignRoot();
this.catalogTracker.waitForRoot();
//we need add this code and guarantee that the transition has completed
this.assignmentManager.waitForAssignment(HRegionInfo.ROOT_REGIONINFO);
assigned++;
}
logs:
2011-08-16 07:45:40,715 DEBUG
[RegionServer:0;C4S2.site,47710,1313495126115-EventThread]
zookeeper.ZooKeeperWatcher(252): regionserver:47710-0x131d2690f780004
Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,715 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
zookeeper.ZKAssign(712): regionserver:47710-0x131d2690f780004 Successfully
transitioned node 70236052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-08-16 07:45:40,715 DEBUG [Thread-760-EventThread]
zookeeper.ZooKeeperWatcher(252): master:60701-0x131d2690f780009 Received
ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,716 INFO [PostOpenDeployTasks:70236052]
catalog.RootLocationEditor(62): Setting ROOT region location in ZooKeeper as
C4S2.site:47710
2011-08-16 07:45:40,716 DEBUG [Thread-760-EventThread]
zookeeper.ZKUtil(1109): master:60701-0x131d2690f780009
[jira] [Commented] (HBASE-4212) TestMasterFailover fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117188#comment-13117188
]
gaojinchao commented on HBASE-4212:
---
I am running all test cases about Trunk. I will give a result tommorrow.
I think Trunk have this problem as well in this case.
1.Master assigns the root
2.Rs set the root address in zk and then crash before finish opening the root.
3.Master get the root data from zk and start assigning the meta region.
4.Root region is offline and needs be ressigned by monitor timeout.
So I think we need wait the root is removed from RIT. It is safe.
TestMasterFailover fails occasionally
-
Key: HBASE-4212
URL: https://issues.apache.org/jira/browse/HBASE-4212
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.4
Reporter: gaojinchao
Assignee: gaojinchao
Fix For: 0.90.5
Attachments: HBASE-4212_TrunkV1.patch, HBASE-4212_branch90V1.patch
It seems a bug. The root in RIT can't be moved..
In the failover process, it enforces root on-line. But not clean zk node.
test will wait forever.
void processFailover() throws KeeperException, IOException,
InterruptedException {
// we enforce on-line root.
HServerInfo hsi =
this.serverManager.getHServerInfo(this.catalogTracker.getMetaLocation());
regionOnline(HRegionInfo.FIRST_META_REGIONINFO, hsi);
hsi =
this.serverManager.getHServerInfo(this.catalogTracker.getRootLocation());
regionOnline(HRegionInfo.ROOT_REGIONINFO, hsi);
It seems that we should wait finished as meta region
int assignRootAndMeta()
throws InterruptedException, IOException, KeeperException {
int assigned = 0;
long timeout = this.conf.getLong(hbase.catalog.verification.timeout,
1000);
// Work on ROOT region. Is it in zk in transition?
boolean rit = this.assignmentManager.
processRegionInTransitionAndBlockUntilAssigned(HRegionInfo.ROOT_REGIONINFO);
if (!catalogTracker.verifyRootRegionLocation(timeout)) {
this.assignmentManager.assignRoot();
this.catalogTracker.waitForRoot();
//we need add this code and guarantee that the transition has completed
this.assignmentManager.waitForAssignment(HRegionInfo.ROOT_REGIONINFO);
assigned++;
}
logs:
2011-08-16 07:45:40,715 DEBUG
[RegionServer:0;C4S2.site,47710,1313495126115-EventThread]
zookeeper.ZooKeeperWatcher(252): regionserver:47710-0x131d2690f780004
Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,715 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
zookeeper.ZKAssign(712): regionserver:47710-0x131d2690f780004 Successfully
transitioned node 70236052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-08-16 07:45:40,715 DEBUG [Thread-760-EventThread]
zookeeper.ZooKeeperWatcher(252): master:60701-0x131d2690f780009 Received
ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,716 INFO [PostOpenDeployTasks:70236052]
catalog.RootLocationEditor(62): Setting ROOT region location in ZooKeeper as
C4S2.site:47710
2011-08-16 07:45:40,716 DEBUG [Thread-760-EventThread]
zookeeper.ZKUtil(1109): master:60701-0x131d2690f780009 Retrieved 52 byte(s)
of data from znode /hbase/unassigned/70236052 and set watcher;
region=-ROOT-,,0, server=C4S2.site,47710,1313495126115,
state=RS_ZK_REGION_OPENING
2011-08-16 07:45:40,717 DEBUG [Thread-760-EventThread]
master.AssignmentManager(477): Handling transition=RS_ZK_REGION_OPENING,
server=C4S2.site,47710,1313495126115, region=70236052/-ROOT-
2011-08-16 07:45:40,725 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
zookeeper.ZKAssign(661): regionserver:47710-0x131d2690f780004 Attempting to
transition node 70236052/-ROOT- from RS_ZK_REGION_OPENING to
RS_ZK_REGION_OPENED
2011-08-16 07:45:40,727 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
zookeeper.ZKUtil(1109): regionserver:47710-0x131d2690f780004 Retrieved 52
byte(s) of data from znode /hbase/unassigned/70236052; data=region=-ROOT-,,0,
server=C4S2.site,47710,1313495126115, state=RS_ZK_REGION_OPENING
2011-08-16 07:45:40,740 DEBUG
[RegionServer:0;C4S2.site,47710,1313495126115-EventThread]
zookeeper.ZooKeeperWatcher(252): regionserver:47710-0x131d2690f780004
Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,740 DEBUG [Thread-760-EventThread]
zookeeper.ZooKeeperWatcher(252): master:60701-0x131d2690f780009 Received
ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,740 DEBUG
[jira] [Commented] (HBASE-4212) TestMasterFailover fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117190#comment-13117190
]
ramkrishna.s.vasudevan commented on HBASE-4212:
---
without the patch i was also able to reproduce the exact problem as Gao reported
TestMasterFailover fails occasionally
-
Key: HBASE-4212
URL: https://issues.apache.org/jira/browse/HBASE-4212
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.4
Reporter: gaojinchao
Assignee: gaojinchao
Fix For: 0.90.5
Attachments: HBASE-4212_TrunkV1.patch, HBASE-4212_branch90V1.patch
It seems a bug. The root in RIT can't be moved..
In the failover process, it enforces root on-line. But not clean zk node.
test will wait forever.
void processFailover() throws KeeperException, IOException,
InterruptedException {
// we enforce on-line root.
HServerInfo hsi =
this.serverManager.getHServerInfo(this.catalogTracker.getMetaLocation());
regionOnline(HRegionInfo.FIRST_META_REGIONINFO, hsi);
hsi =
this.serverManager.getHServerInfo(this.catalogTracker.getRootLocation());
regionOnline(HRegionInfo.ROOT_REGIONINFO, hsi);
It seems that we should wait finished as meta region
int assignRootAndMeta()
throws InterruptedException, IOException, KeeperException {
int assigned = 0;
long timeout = this.conf.getLong(hbase.catalog.verification.timeout,
1000);
// Work on ROOT region. Is it in zk in transition?
boolean rit = this.assignmentManager.
processRegionInTransitionAndBlockUntilAssigned(HRegionInfo.ROOT_REGIONINFO);
if (!catalogTracker.verifyRootRegionLocation(timeout)) {
this.assignmentManager.assignRoot();
this.catalogTracker.waitForRoot();
//we need add this code and guarantee that the transition has completed
this.assignmentManager.waitForAssignment(HRegionInfo.ROOT_REGIONINFO);
assigned++;
}
logs:
2011-08-16 07:45:40,715 DEBUG
[RegionServer:0;C4S2.site,47710,1313495126115-EventThread]
zookeeper.ZooKeeperWatcher(252): regionserver:47710-0x131d2690f780004
Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,715 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
zookeeper.ZKAssign(712): regionserver:47710-0x131d2690f780004 Successfully
transitioned node 70236052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-08-16 07:45:40,715 DEBUG [Thread-760-EventThread]
zookeeper.ZooKeeperWatcher(252): master:60701-0x131d2690f780009 Received
ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,716 INFO [PostOpenDeployTasks:70236052]
catalog.RootLocationEditor(62): Setting ROOT region location in ZooKeeper as
C4S2.site:47710
2011-08-16 07:45:40,716 DEBUG [Thread-760-EventThread]
zookeeper.ZKUtil(1109): master:60701-0x131d2690f780009 Retrieved 52 byte(s)
of data from znode /hbase/unassigned/70236052 and set watcher;
region=-ROOT-,,0, server=C4S2.site,47710,1313495126115,
state=RS_ZK_REGION_OPENING
2011-08-16 07:45:40,717 DEBUG [Thread-760-EventThread]
master.AssignmentManager(477): Handling transition=RS_ZK_REGION_OPENING,
server=C4S2.site,47710,1313495126115, region=70236052/-ROOT-
2011-08-16 07:45:40,725 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
zookeeper.ZKAssign(661): regionserver:47710-0x131d2690f780004 Attempting to
transition node 70236052/-ROOT- from RS_ZK_REGION_OPENING to
RS_ZK_REGION_OPENED
2011-08-16 07:45:40,727 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
zookeeper.ZKUtil(1109): regionserver:47710-0x131d2690f780004 Retrieved 52
byte(s) of data from znode /hbase/unassigned/70236052; data=region=-ROOT-,,0,
server=C4S2.site,47710,1313495126115, state=RS_ZK_REGION_OPENING
2011-08-16 07:45:40,740 DEBUG
[RegionServer:0;C4S2.site,47710,1313495126115-EventThread]
zookeeper.ZooKeeperWatcher(252): regionserver:47710-0x131d2690f780004
Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,740 DEBUG [Thread-760-EventThread]
zookeeper.ZooKeeperWatcher(252): master:60701-0x131d2690f780009 Received
ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,740 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
zookeeper.ZKAssign(712): regionserver:47710-0x131d2690f780004 Successfully
transitioned node 70236052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
2011-08-16 07:45:40,741 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
handler.OpenRegionHandler(121): Opened -ROOT-,,0.70236052
2011-08-16
[jira] [Commented] (HBASE-4454) Add failsafe plugin to build and rename integration tests
[ https://issues.apache.org/jira/browse/HBASE-4454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117194#comment-13117194 ] Hudson commented on HBASE-4454: --- Integrated in HBase-0.92 #27 (See [https://builds.apache.org/job/HBase-0.92/27/]) HBASE-4454 Add failsafe plugin to build and rename integration tests stack : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/pom.xml Add failsafe plugin to build and rename integration tests - Key: HBASE-4454 URL: https://issues.apache.org/jira/browse/HBASE-4454 Project: HBase Issue Type: Improvement Reporter: Jesse Yates Fix For: 0.92.0 Attachments: mvn_HBASE-4454.patch Add the maven-failsafe-plugin to the build process so we can run integration tests with mvn verify. This will also involve a renaming of integration tests to conform to a new integration test regex. This is a stopgap measure while we until break them out into their own module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4492) TestRollingRestart fails intermittently
[
https://issues.apache.org/jira/browse/HBASE-4492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117230#comment-13117230
]
Hudson commented on HBASE-4492:
---
Integrated in HBase-TRUNK #2268 (See
[https://builds.apache.org/job/HBase-TRUNK/2268/])
HBASE-4492 TestRollingRestart Fails Intermittently(Ted yu and Ram)
ramkrishna :
Files :
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
*
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestRollingRestart.java
TestRollingRestart fails intermittently
---
Key: HBASE-4492
URL: https://issues.apache.org/jira/browse/HBASE-4492
Project: HBase
Issue Type: Test
Affects Versions: 0.92.0
Reporter: Ted Yu
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.0
Attachments: 4492-v2.txt, 4492.txt, HBASE-4492.patch
I got the following when running test suite on TRUNK:
{code}
testBasicRollingRestart(org.apache.hadoop.hbase.master.TestRollingRestart)
Time elapsed: 300.28 sec ERROR!
java.lang.Exception: test timed out after 30 milliseconds
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.master.TestRollingRestart.waitForRSShutdownToStartAndFinish(TestRollingRestart.java:313)
at
org.apache.hadoop.hbase.master.TestRollingRestart.testBasicRollingRestart(TestRollingRestart.java:210)
{code}
I ran TestRollingRestart#testBasicRollingRestart manually afterwards which
wiped out test output file for the failed test.
Similar failure can be found on Jenkins:
https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/19/testReport/junit/org.apache.hadoop.hbase.master/TestRollingRestart/testBasicRollingRestart/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4212) TestMasterFailover fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117292#comment-13117292
]
Ted Yu commented on HBASE-4212:
---
Now that HBASE-4511 has been opened, I vote +1 on Jinchao's patch.
TestMasterFailover fails occasionally
-
Key: HBASE-4212
URL: https://issues.apache.org/jira/browse/HBASE-4212
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.4
Reporter: gaojinchao
Assignee: gaojinchao
Fix For: 0.90.5
Attachments: HBASE-4212_TrunkV1.patch, HBASE-4212_branch90V1.patch
It seems a bug. The root in RIT can't be moved..
In the failover process, it enforces root on-line. But not clean zk node.
test will wait forever.
void processFailover() throws KeeperException, IOException,
InterruptedException {
// we enforce on-line root.
HServerInfo hsi =
this.serverManager.getHServerInfo(this.catalogTracker.getMetaLocation());
regionOnline(HRegionInfo.FIRST_META_REGIONINFO, hsi);
hsi =
this.serverManager.getHServerInfo(this.catalogTracker.getRootLocation());
regionOnline(HRegionInfo.ROOT_REGIONINFO, hsi);
It seems that we should wait finished as meta region
int assignRootAndMeta()
throws InterruptedException, IOException, KeeperException {
int assigned = 0;
long timeout = this.conf.getLong(hbase.catalog.verification.timeout,
1000);
// Work on ROOT region. Is it in zk in transition?
boolean rit = this.assignmentManager.
processRegionInTransitionAndBlockUntilAssigned(HRegionInfo.ROOT_REGIONINFO);
if (!catalogTracker.verifyRootRegionLocation(timeout)) {
this.assignmentManager.assignRoot();
this.catalogTracker.waitForRoot();
//we need add this code and guarantee that the transition has completed
this.assignmentManager.waitForAssignment(HRegionInfo.ROOT_REGIONINFO);
assigned++;
}
logs:
2011-08-16 07:45:40,715 DEBUG
[RegionServer:0;C4S2.site,47710,1313495126115-EventThread]
zookeeper.ZooKeeperWatcher(252): regionserver:47710-0x131d2690f780004
Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,715 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
zookeeper.ZKAssign(712): regionserver:47710-0x131d2690f780004 Successfully
transitioned node 70236052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-08-16 07:45:40,715 DEBUG [Thread-760-EventThread]
zookeeper.ZooKeeperWatcher(252): master:60701-0x131d2690f780009 Received
ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,716 INFO [PostOpenDeployTasks:70236052]
catalog.RootLocationEditor(62): Setting ROOT region location in ZooKeeper as
C4S2.site:47710
2011-08-16 07:45:40,716 DEBUG [Thread-760-EventThread]
zookeeper.ZKUtil(1109): master:60701-0x131d2690f780009 Retrieved 52 byte(s)
of data from znode /hbase/unassigned/70236052 and set watcher;
region=-ROOT-,,0, server=C4S2.site,47710,1313495126115,
state=RS_ZK_REGION_OPENING
2011-08-16 07:45:40,717 DEBUG [Thread-760-EventThread]
master.AssignmentManager(477): Handling transition=RS_ZK_REGION_OPENING,
server=C4S2.site,47710,1313495126115, region=70236052/-ROOT-
2011-08-16 07:45:40,725 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
zookeeper.ZKAssign(661): regionserver:47710-0x131d2690f780004 Attempting to
transition node 70236052/-ROOT- from RS_ZK_REGION_OPENING to
RS_ZK_REGION_OPENED
2011-08-16 07:45:40,727 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
zookeeper.ZKUtil(1109): regionserver:47710-0x131d2690f780004 Retrieved 52
byte(s) of data from znode /hbase/unassigned/70236052; data=region=-ROOT-,,0,
server=C4S2.site,47710,1313495126115, state=RS_ZK_REGION_OPENING
2011-08-16 07:45:40,740 DEBUG
[RegionServer:0;C4S2.site,47710,1313495126115-EventThread]
zookeeper.ZooKeeperWatcher(252): regionserver:47710-0x131d2690f780004
Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,740 DEBUG [Thread-760-EventThread]
zookeeper.ZooKeeperWatcher(252): master:60701-0x131d2690f780009 Received
ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/unassigned/70236052
2011-08-16 07:45:40,740 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
zookeeper.ZKAssign(712): regionserver:47710-0x131d2690f780004 Successfully
transitioned node 70236052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
2011-08-16 07:45:40,741 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0]
handler.OpenRegionHandler(121): Opened -ROOT-,,0.70236052
2011-08-16 07:45:40,741 DEBUG [Thread-760-EventThread]
[jira] [Updated] (HBASE-4508) Backport HBASE-3777 to 0.90 branch
[ https://issues.apache.org/jira/browse/HBASE-4508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bright Fulton updated HBASE-4508: - Attachment: HBASE-4508.v1.patch This is HBASE-3777.v8.patch backported to 0.90 at rev 1175062 and with the addition of the hbase.client.per.config.inst config param. Backport HBASE-3777 to 0.90 branch -- Key: HBASE-4508 URL: https://issues.apache.org/jira/browse/HBASE-4508 Project: HBase Issue Type: Bug Reporter: Ted Yu Attachments: HBASE-4508.v1.patch See discussion here: http://search-hadoop.com/m/MJBId1aazTR1/backporting+HBASE-3777+to+0.90subj=backporting+HBASE+3777+to+0+90 Rocketfuel has been running 0.90.3 with HBASE-3777 since its resolution. They have 10 RS nodes , 1 Master and 1 Zookeeper Live writes and reads but super heavy on reads. Cache hit is pretty high. The qps on one of their data centers is 50K. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4145) Provide metrics for hbase client
[ https://issues.apache.org/jira/browse/HBASE-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117295#comment-13117295 ] jirapos...@reviews.apache.org commented on HBASE-4145: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1674/#review2150 --- http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java https://reviews.apache.org/r/1674/#comment4994 I think we need not give copyright information. - ramkrishna On 2011-09-28 23:03:54, Ming Ma wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1674/ bq. --- bq. bq. (Updated 2011-09-28 23:03:54) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. 1. Collect client-side scan related metrics during scan operation. It is turned off by default. bq. 2. TableInputFormat enables metrics collection on scan and pass the data to mapreduce framework. It only works with new mapreduce APIs that allow TableInputFormat to get access to mapreduce Counter. bq. 3. Clean up some minor issues in tableInputFormat as well as test code. bq. bq. bq. This addresses bug hbase-4145. bq. https://issues.apache.org/jira/browse/hbase-4145 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReader.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java 1176942 bq. bq. Diff: https://reviews.apache.org/r/1674/diff bq. bq. bq. Testing bq. --- bq. bq. 1. Verified on a small cluster. bq. 2. Existing unit tests. bq. 3. Added new tests. bq. bq. bq. Thanks, bq. bq. Ming bq. bq. Provide metrics for hbase client Key: HBASE-4145 URL: https://issues.apache.org/jira/browse/HBASE-4145 Project: HBase Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma Attachments: HBaseClientSideMetrics.jpg Sometimes it is useful to get some metrics from hbase client point of view. This will help understand the metrics for scan/TableInputFormat map job scenario. What to capture, for example, for each ResultScanner object, 1. The number of RPC calls to RSs. 2. The delta time between consecutive RPC calls in the current serialized scan implementation. 3. The number of RPC retry to RSs. 4. The number of NotServingRegionException got. 5. The number of remote RPC calls. This excludes those call that hbase client calls the RS on the same machine. 6. The number of regions accessed. How to capture 1. Metrics framework works for a fixed number of metrics. It doesn't fit this scenario. 2. Use some TBD solution in HBase to capture such dynamic metrics. If we assume there is a solution in HBase that HBase client can use to log such kind of metrics, TableInputFormat can pass in mapreduce task ID as application scan ID to HBase client as small addition to existing scan API; and HBase client can log metrics accordingly with such ID. That will allow query, analysis later on the metrics data for specific map reduce job. 3. Expose via MapReduce counter. It lacks certain features, for example, there is no good way to access the metrics on per map instance; the MapReduce framework only performs sum on the counter values so it is tricky to find the max of certain metrics in
[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog
[
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117298#comment-13117298
]
Ted Yu commented on HBASE-4487:
---
I got the following from test suite run:
{code}
testAppend(org.apache.hadoop.hbase.regionserver.wal.TestHLog) Time elapsed:
0.037 sec ERROR!
java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.wal.TestHLog.testAppend(TestHLog.java:567)
{code}
Please use my script on HBASE-4480 to reproduce the above.
I can provide test output if needed.
The increment operation can release the rowlock before sync-ing the Hlog
Key: HBASE-4487
URL: https://issues.apache.org/jira/browse/HBASE-4487
Project: HBase
Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
Attachments: appendNoSync4.txt, appendNoSync5.txt
This allows for better throughput when there are hot rows.I have seen this
change make a single row update improve from 400 increments/sec/server to
4000 increments/sec/server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4492) TestRollingRestart fails intermittently
[
https://issues.apache.org/jira/browse/HBASE-4492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117344#comment-13117344
]
Hudson commented on HBASE-4492:
---
Integrated in HBase-0.92 #28 (See
[https://builds.apache.org/job/HBase-0.92/28/])
HBASE-4492 TestMasterFailover fails intermittently(Ted yu and Ram)
ramkrishna :
Files :
* /hbase/branches/0.92/CHANGES.txt
*
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
*
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestRollingRestart.java
TestRollingRestart fails intermittently
---
Key: HBASE-4492
URL: https://issues.apache.org/jira/browse/HBASE-4492
Project: HBase
Issue Type: Test
Affects Versions: 0.92.0
Reporter: Ted Yu
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.0
Attachments: 4492-v2.txt, 4492.txt, HBASE-4492.patch
I got the following when running test suite on TRUNK:
{code}
testBasicRollingRestart(org.apache.hadoop.hbase.master.TestRollingRestart)
Time elapsed: 300.28 sec ERROR!
java.lang.Exception: test timed out after 30 milliseconds
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.master.TestRollingRestart.waitForRSShutdownToStartAndFinish(TestRollingRestart.java:313)
at
org.apache.hadoop.hbase.master.TestRollingRestart.testBasicRollingRestart(TestRollingRestart.java:210)
{code}
I ran TestRollingRestart#testBasicRollingRestart manually afterwards which
wiped out test output file for the failed test.
Similar failure can be found on Jenkins:
https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/19/testReport/junit/org.apache.hadoop.hbase.master/TestRollingRestart/testBasicRollingRestart/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4385) Use CacheBuilder in place of MapMaker
[ https://issues.apache.org/jira/browse/HBASE-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117411#comment-13117411 ] Ted Yu commented on HBASE-4385: --- Since guava r10 has been released, I tried to create a patch by using CacheBuilder last night. It turned out that r10 supports the transition from ComputingMap at the moment. Our use case isn't supported yet. I will revisit this when proper support is added in future guava release. Use CacheBuilder in place of MapMaker - Key: HBASE-4385 URL: https://issues.apache.org/jira/browse/HBASE-4385 Project: HBase Issue Type: Task Reporter: Ted Yu Guava release 10 introduced CacheBuilder. We should use it in place of MapMaker which is used for caching. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4512) JVMClusterUtil throwing wrong exception when master thread cannot be created
JVMClusterUtil throwing wrong exception when master thread cannot be created
Key: HBASE-4512
URL: https://issues.apache.org/jira/browse/HBASE-4512
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Trivial
In JVMClusterUtil.MasterThread createMasterThread
{code}
throw new RuntimeException(Failed construction of RegionServer: +
{code}
It must be failed construction of Master.
Very trivial one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4385) Use CacheBuilder in place of MapMaker
[ https://issues.apache.org/jira/browse/HBASE-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117417#comment-13117417 ] Li Pi commented on HBASE-4385: -- Awesome! I can take care of it. The evictionlistener of CacheBuilder is slightly different. On Thu, Sep 29, 2011 at 9:36 AM, Ted Yu (Commented) (JIRA) Use CacheBuilder in place of MapMaker - Key: HBASE-4385 URL: https://issues.apache.org/jira/browse/HBASE-4385 Project: HBase Issue Type: Task Reporter: Ted Yu Guava release 10 introduced CacheBuilder. We should use it in place of MapMaker which is used for caching. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4512) JVMClusterUtil throwing wrong exception when master thread cannot be created
[
https://issues.apache.org/jira/browse/HBASE-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-4512:
--
Attachment: HBASE_4253_0.90.patch
for 0.90
JVMClusterUtil throwing wrong exception when master thread cannot be created
Key: HBASE-4512
URL: https://issues.apache.org/jira/browse/HBASE-4512
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Trivial
Attachments: HBASE-4512_trunk.patch, HBASE_4253_0.90.patch
In JVMClusterUtil.MasterThread createMasterThread
{code}
throw new RuntimeException(Failed construction of RegionServer: +
{code}
It must be failed construction of Master.
Very trivial one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4512) JVMClusterUtil throwing wrong exception when master thread cannot be created
[
https://issues.apache.org/jira/browse/HBASE-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-4512:
--
Attachment: HBASE-4512_trunk.patch
For trunk
JVMClusterUtil throwing wrong exception when master thread cannot be created
Key: HBASE-4512
URL: https://issues.apache.org/jira/browse/HBASE-4512
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Trivial
Attachments: HBASE-4512_trunk.patch, HBASE_4253_0.90.patch
In JVMClusterUtil.MasterThread createMasterThread
{code}
throw new RuntimeException(Failed construction of RegionServer: +
{code}
It must be failed construction of Master.
Very trivial one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4512) JVMClusterUtil throwing wrong exception when master thread cannot be created
[
https://issues.apache.org/jira/browse/HBASE-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117420#comment-13117420
]
ramkrishna.s.vasudevan commented on HBASE-4512:
---
If you guys are ok with this I can commit this.
JVMClusterUtil throwing wrong exception when master thread cannot be created
Key: HBASE-4512
URL: https://issues.apache.org/jira/browse/HBASE-4512
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Trivial
Attachments: HBASE-4512_trunk.patch, HBASE_4253_0.90.patch
In JVMClusterUtil.MasterThread createMasterThread
{code}
throw new RuntimeException(Failed construction of RegionServer: +
{code}
It must be failed construction of Master.
Very trivial one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4385) Use CacheBuilder in place of MapMaker
[ https://issues.apache.org/jira/browse/HBASE-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117422#comment-13117422 ] Li Pi commented on HBASE-4385: -- The removallistener vs evictionlistener gets rid of a possible source of race conditions, I wanted to use cachebuilder initially, but it wasn't available. Use CacheBuilder in place of MapMaker - Key: HBASE-4385 URL: https://issues.apache.org/jira/browse/HBASE-4385 Project: HBase Issue Type: Task Reporter: Ted Yu Guava release 10 introduced CacheBuilder. We should use it in place of MapMaker which is used for caching. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store metadata into the transaction log
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117428#comment-13117428 ] dhruba borthakur commented on HBASE-4477: - I ran all unit tests. All unit tests pass, except these: TestDistributedLogSplitting, TestRollingRestart, TestHTablePool. And these failures are not related to my patch. Andrew: will you be able to review this patch, especially because it changes the coprocessor interface? Ability for an application to store metadata into the transaction log - Key: HBASE-4477 URL: https://issues.apache.org/jira/browse/HBASE-4477 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: coprocessorPut1.txt, hlogMetadata1.txt mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE. The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4385) Use CacheBuilder in place of MapMaker
[ https://issues.apache.org/jira/browse/HBASE-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4385: -- Attachment: 4385.txt Here is unfinished work, just for reference. Use CacheBuilder in place of MapMaker - Key: HBASE-4385 URL: https://issues.apache.org/jira/browse/HBASE-4385 Project: HBase Issue Type: Task Reporter: Ted Yu Attachments: 4385.txt Guava release 10 introduced CacheBuilder. We should use it in place of MapMaker which is used for caching. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4512) JVMClusterUtil throwing wrong exception when master thread cannot be created
[
https://issues.apache.org/jira/browse/HBASE-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117436#comment-13117436
]
Ted Yu commented on HBASE-4512:
---
+1 on patch(es).
JVMClusterUtil throwing wrong exception when master thread cannot be created
Key: HBASE-4512
URL: https://issues.apache.org/jira/browse/HBASE-4512
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Trivial
Attachments: HBASE-4512_trunk.patch, HBASE_4253_0.90.patch
In JVMClusterUtil.MasterThread createMasterThread
{code}
throw new RuntimeException(Failed construction of RegionServer: +
{code}
It must be failed construction of Master.
Very trivial one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store metadata into the transaction log
[
https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117441#comment-13117441
]
Todd Lipcon commented on HBASE-4477:
Just a thought while we're changing the APIs: might make sense to do something
like introduce the following interface:
{code}
interface CPPutInfo {
public Put getPutObject();
public boolean getWriteToWAL();
public WALEdit getWalEdit();
}
{code}
then, if we want to extend this API with more info about the put later, we can
simply add more methods and not break compatibility? We should probably do this
in another JIRA, and for all of the RegionObserver hooks - this JIRA just gave
me the idea.
Ability for an application to store metadata into the transaction log
-
Key: HBASE-4477
URL: https://issues.apache.org/jira/browse/HBASE-4477
Project: HBase
Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
Attachments: coprocessorPut1.txt, hlogMetadata1.txt
mySQL allows an application to store an arbitrary blob along with each
transaction in its transaction logs. This JIRA is to have a similar feature
request for HBASE.
The use case is as follows: An application on one data center A stores a blob
of data along with each transaction. A replication software picks up these
blobs from the transaction logs in A and hands it to another instance of the
same application running on a remote data center B. The application in B is
responsible for applying this to the remote Hbase cluster (and also handle
conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store metadata into the transaction log
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117446#comment-13117446 ] Ted Yu commented on HBASE-4477: --- I second Todd's idea. Should CPPutInfo be created in this JIRA and we can work on other parts in another JIRA ? Ability for an application to store metadata into the transaction log - Key: HBASE-4477 URL: https://issues.apache.org/jira/browse/HBASE-4477 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: coprocessorPut1.txt, hlogMetadata1.txt mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE. The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store metadata into the transaction log
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117449#comment-13117449 ] Jonathan Gray commented on HBASE-4477: -- +1 on CPPutInfo, CPGetInfo, etc... Ability for an application to store metadata into the transaction log - Key: HBASE-4477 URL: https://issues.apache.org/jira/browse/HBASE-4477 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: coprocessorPut1.txt, hlogMetadata1.txt mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE. The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store metadata into the transaction log
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117450#comment-13117450 ] Jonathan Gray commented on HBASE-4477: -- And yeah, maybe introduce CPPutInfo in this JIRA and open a follow-up to change the others Ability for an application to store metadata into the transaction log - Key: HBASE-4477 URL: https://issues.apache.org/jira/browse/HBASE-4477 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: coprocessorPut1.txt, hlogMetadata1.txt mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE. The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store metadata into the transaction log
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117451#comment-13117451 ] dhruba borthakur commented on HBASE-4477: - Hi Todd, I like your idea, and I will create another jira for that one. It will have to touch each and every coprocessor api. (If you are ok with it, I would like this jira to make the prePut and preGet be similar in the sense that they both take the Put/get object). Ability for an application to store metadata into the transaction log - Key: HBASE-4477 URL: https://issues.apache.org/jira/browse/HBASE-4477 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: coprocessorPut1.txt, hlogMetadata1.txt mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE. The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4512) JVMClusterUtil throwing wrong exception when master thread cannot be created
[
https://issues.apache.org/jira/browse/HBASE-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117454#comment-13117454
]
ramkrishna.s.vasudevan commented on HBASE-4512:
---
Integrated to 0.90, trunk and 0.92 branch.
Thanks Ted for your review.
JVMClusterUtil throwing wrong exception when master thread cannot be created
Key: HBASE-4512
URL: https://issues.apache.org/jira/browse/HBASE-4512
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Trivial
Attachments: HBASE-4512_trunk.patch, HBASE_4253_0.90.patch
In JVMClusterUtil.MasterThread createMasterThread
{code}
throw new RuntimeException(Failed construction of RegionServer: +
{code}
It must be failed construction of Master.
Very trivial one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4512) JVMClusterUtil throwing wrong exception when master thread cannot be created
[
https://issues.apache.org/jira/browse/HBASE-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-4512:
--
Fix Version/s: 0.90.5
0.94.0
0.92.0
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
JVMClusterUtil throwing wrong exception when master thread cannot be created
Key: HBASE-4512
URL: https://issues.apache.org/jira/browse/HBASE-4512
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Trivial
Fix For: 0.92.0, 0.94.0, 0.90.5
Attachments: HBASE-4512_trunk.patch, HBASE_4253_0.90.patch
In JVMClusterUtil.MasterThread createMasterThread
{code}
throw new RuntimeException(Failed construction of RegionServer: +
{code}
It must be failed construction of Master.
Very trivial one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class
[ https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117456#comment-13117456 ] jirapos...@reviews.apache.org commented on HBASE-4422: -- bq. On 2011-09-29 00:50:58, Andrew Purtell wrote: bq. /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java, line 112 bq. https://reviews.apache.org/r/2089/diff/1/?file=46305#file46305line112 bq. bq. CacheConfig could extend Configuration? It's only extra constants and constructors, really. I considered doing that. But they seem orthogonal in most of their usage and I think it'd be weird to pass our Configuration all the way down into HFile reading and such. Changing config values dynamically (like disabling caching for a specific read) would then require changing the base Configuration or would the local booleans in the CacheConfig not match what's in the underlying Configuration? - Jonathan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2089/#review2145 --- On 2011-09-28 19:56:14, Jonathan Gray wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2089/ bq. --- bq. bq. (Updated 2011-09-28 19:56:14) bq. bq. bq. Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Creates a new CacheConfig class and moves almost everything block cache related into this single class. Adding new configuration params and booleans and such should be much better. bq. bq. All tests are NOT passing yet, still working on it, but wanted to have something up today. Basically code complete but broken :) bq. bq. bq. This addresses bug HBASE-4422. bq. https://issues.apache.org/jira/browse/HBASE-4422 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java PRE-CREATION bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 1177030 bq. /src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 1177030 bq. /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1177030 bq. /src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java 1177030 bq.
[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class
[ https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117458#comment-13117458 ] jirapos...@reviews.apache.org commented on HBASE-4422: -- bq. On 2011-09-29 01:10:28, Lars Hofhansl wrote: bq. Cool... This makes it much simpler in the future. bq. Are you still planning to fold HBASE-4496 into this? Yes, I will look at that today. I'm thinking about a new test for all this stuff since this diff is riddled with bugs and it's a bit random which tests fail :) bq. On 2011-09-29 01:10:28, Lars Hofhansl wrote: bq. /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java, line 244 bq. https://reviews.apache.org/r/2089/diff/1/?file=46310#file46310line244 bq. bq. Hmm... The new code does not honor the cacheBlock flag that was passed into the method. Yeah, this is the stuff I need to go back and review. Thanks for pointing me to the right place :) Will have a new diff up in a bit. - Jonathan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2089/#review2147 --- On 2011-09-28 19:56:14, Jonathan Gray wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2089/ bq. --- bq. bq. (Updated 2011-09-28 19:56:14) bq. bq. bq. Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Creates a new CacheConfig class and moves almost everything block cache related into this single class. Adding new configuration params and booleans and such should be much better. bq. bq. All tests are NOT passing yet, still working on it, but wanted to have something up today. Basically code complete but broken :) bq. bq. bq. This addresses bug HBASE-4422. bq. https://issues.apache.org/jira/browse/HBASE-4422 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java PRE-CREATION bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 1177030 bq. /src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 1177030 bq. /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1177030 bq. /src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java 1177030 bq.
[jira] [Commented] (HBASE-4477) Ability for an application to store metadata into the transaction log
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117459#comment-13117459 ] Todd Lipcon commented on HBASE-4477: Sure, I don't particularly care if it happens as part of this JIRA or separately. Just wanted to drop my 2 cents :) Ability for an application to store metadata into the transaction log - Key: HBASE-4477 URL: https://issues.apache.org/jira/browse/HBASE-4477 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: coprocessorPut1.txt, hlogMetadata1.txt mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE. The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4454) Add failsafe plugin to build and rename integration tests
[ https://issues.apache.org/jira/browse/HBASE-4454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117465#comment-13117465 ] Jesse Yates commented on HBASE-4454: @stack - no worries, whenever you get around to it. Thanks for updating the book! Add failsafe plugin to build and rename integration tests - Key: HBASE-4454 URL: https://issues.apache.org/jira/browse/HBASE-4454 Project: HBase Issue Type: Improvement Reporter: Jesse Yates Fix For: 0.92.0 Attachments: mvn_HBASE-4454.patch Add the maven-failsafe-plugin to the build process so we can run integration tests with mvn verify. This will also involve a renaming of integration tests to conform to a new integration test regex. This is a stopgap measure while we until break them out into their own module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4513) NOTICES.txt refers to Facebook for Thrift
NOTICES.txt refers to Facebook for Thrift
-
Key: HBASE-4513
URL: https://issues.apache.org/jira/browse/HBASE-4513
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Joe Pallas
Priority: Minor
The NOTICES.txt file says:
{quote}
In addition, this product includes software developed by:
Facebook, Inc. (http://developers.facebook.com/thrift/ -- Page includes the
Thrift Software License)
{quote}
Since Thrift is now an Apache project, this seems to be obsolete.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store metadata into the transaction log
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117472#comment-13117472 ] Gary Helmling commented on HBASE-4477: -- @Todd, I like the idea of using XXXInfo objects to insulate coprocessors from API changes they're not using. @Dhruba, Agree that it would be better to tackle that in a separate JIRA and keep all of the APIs in sync. Adopting just PutInfo here would leave us with an oddly inconsistent API after this patch. @All, Can we drop the CP prefix and just use PutInfo, GetInfo, DeleteInfo, etc? It looks ugly to me and unnecessary if we place these classes under o.a.h.h.coprocessor for namespacing. Also, I like that this makes the exposed Observer APIs less brittle, but it does add a fair amount of complexity to the APIs in new classes. To carry this through, we really need a XXXInfo class per operation... Ability for an application to store metadata into the transaction log - Key: HBASE-4477 URL: https://issues.apache.org/jira/browse/HBASE-4477 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: coprocessorPut1.txt, hlogMetadata1.txt mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE. The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store metadata into the transaction log
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117479#comment-13117479 ] Ted Yu commented on HBASE-4477: --- Todd didn't mention which namespace CPPutInfo should be placed. If we cannot foresee that other components would utilize PutInfo, we should place PutInfo in o.a.h.h.coprocessor without the CP prefix. Ability for an application to store metadata into the transaction log - Key: HBASE-4477 URL: https://issues.apache.org/jira/browse/HBASE-4477 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: coprocessorPut1.txt, hlogMetadata1.txt mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE. The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4513) NOTICES.txt refers to Facebook for Thrift
[
https://issues.apache.org/jira/browse/HBASE-4513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117491#comment-13117491
]
stack commented on HBASE-4513:
--
I'm committing this:
{code}
Index: NOTICE.txt
===
--- NOTICE.txt (revision 1176516)
+++ NOTICE.txt (working copy)
@@ -3,8 +3,6 @@
In addition, this product includes software developed by:
-Facebook, Inc. (http://developers.facebook.com/thrift/ -- Page includes the
Thrift Software License)
-
JUnit (http://www.junit.org/) included under the Common Public License v1.0.
See
the full text here: http://junit.sourceforge.net/cpl-v10.html
{code}
NOTICES.txt refers to Facebook for Thrift
-
Key: HBASE-4513
URL: https://issues.apache.org/jira/browse/HBASE-4513
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Joe Pallas
Priority: Minor
The NOTICES.txt file says:
{quote}
In addition, this product includes software developed by:
Facebook, Inc. (http://developers.facebook.com/thrift/ -- Page includes the
Thrift Software License)
{quote}
Since Thrift is now an Apache project, this seems to be obsolete.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4513) NOTICES.txt refers to Facebook for Thrift
[
https://issues.apache.org/jira/browse/HBASE-4513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-4513.
--
Resolution: Fixed
Fix Version/s: 0.92.0
Assignee: stack
Committed to 0.92 branch and TRUNK.
NOTICES.txt refers to Facebook for Thrift
-
Key: HBASE-4513
URL: https://issues.apache.org/jira/browse/HBASE-4513
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Joe Pallas
Assignee: stack
Priority: Minor
Fix For: 0.92.0
The NOTICES.txt file says:
{quote}
In addition, this product includes software developed by:
Facebook, Inc. (http://developers.facebook.com/thrift/ -- Page includes the
Thrift Software License)
{quote}
Since Thrift is now an Apache project, this seems to be obsolete.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store metadata into the transaction log
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117496#comment-13117496 ] Jonathan Gray commented on HBASE-4477: -- PutInfo seems overly generic but I agree that CPPutInfo is straight ugly. And I keep thinking it says CPUInfo. So Dhruba should just extend the API for now and we can introduce these new classes in a follow-up jira. Ability for an application to store metadata into the transaction log - Key: HBASE-4477 URL: https://issues.apache.org/jira/browse/HBASE-4477 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: coprocessorPut1.txt, hlogMetadata1.txt mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE. The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store metadata into the transaction log
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117497#comment-13117497 ] Jonathan Gray commented on HBASE-4477: -- And it looks like the patch from this morning does exactly that. I'm +1 on coprocessorPut1.txt. Someone else want to review? Ability for an application to store metadata into the transaction log - Key: HBASE-4477 URL: https://issues.apache.org/jira/browse/HBASE-4477 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: coprocessorPut1.txt, coprocessorPut2.txt, hlogMetadata1.txt mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE. The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4477) Ability for an application to store metadata into the transaction log
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-4477: -- Attachment: coprocessorPut2.txt +1 Attached updated patch 'coprocessorPut2.txt which removes some unused locals. I'm curious if passing the cluster ID to internalPut and internalDelete is necessary? They can be retrieved from the Put or Delete, or the Put or Delete can be modified. I don't know enough about replication there: The put or delete gets written to the WAL with one UUID but internally contains another? Ability for an application to store metadata into the transaction log - Key: HBASE-4477 URL: https://issues.apache.org/jira/browse/HBASE-4477 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: coprocessorPut1.txt, coprocessorPut2.txt, hlogMetadata1.txt mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE. The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store metadata into the transaction log
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117500#comment-13117500 ] Andrew Purtell commented on HBASE-4477: --- In my opinion prefixing anything regarding coprocessors with CP should be an anti-pattern. Ability for an application to store metadata into the transaction log - Key: HBASE-4477 URL: https://issues.apache.org/jira/browse/HBASE-4477 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: coprocessorPut1.txt, coprocessorPut2.txt, hlogMetadata1.txt mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE. The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store metadata into the transaction log
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117507#comment-13117507 ] Gary Helmling commented on HBASE-4477: -- +1 on coprocessorPut2.txt, which just removes the unused familyMap local vars in RegionCoprocessorHost. I'm curious about the cluster ID question as well though. Don't know enough about the replication stuff to understand the current usage. Ability for an application to store metadata into the transaction log - Key: HBASE-4477 URL: https://issues.apache.org/jira/browse/HBASE-4477 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: coprocessorPut1.txt, coprocessorPut2.txt, hlogMetadata1.txt mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE. The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2794) ROWCOL bloom filter not used if multiple columns within same family are requested in a Get
[
https://issues.apache.org/jira/browse/HBASE-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117517#comment-13117517
]
jirapos...@reviews.apache.org commented on HBASE-2794:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2084/#review2161
---
src/main/java/org/apache/hadoop/hbase/KeyValue.java
https://reviews.apache.org/r/2084/#comment5035
I was implying that this is also a method argument when I wrote this
comment. I will edit this to make it clearer.
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
https://reviews.apache.org/r/2084/#comment5036
Yes, I will modify the javadoc of this method.
- Mikhail
On 2011-09-28 16:03:52, Mikhail Bautin wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2084/
bq. ---
bq.
bq. (Updated 2011-09-28 16:03:52)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Previously we only used row-column Bloom filters for scans that only
requested one column. We have seen production queries that request up to 200
columns, and with say ~6 store files per store (region / column family
combination) this might have resulted in 1200 block read operations in the
worst case. With this diff we will be avoiding seeks on store files that we
know don't contain the row/column of interest when using an
ExplicitColumnTracker. The performance should remain the same for column range
queries.
bq.
bq.
bq. This addresses bug HBASE-2794.
bq. https://issues.apache.org/jira/browse/HBASE-2794
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
08d3ba4
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
ac2348e
bq.src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 4aa72de
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
68cdac5
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
fd9e7ef
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
9d9895c
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
6cdada7
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7cbdb98
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/KeyValue.java 585c4a8
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
f5173c4
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java a3d778e
bq.src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java
32f88fb
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueHeap.java
a5d13f7
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java
baee696
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/2084/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Existing unit tests. A new unit test (TestScanWithBloomError). Load
testing using HBaseTest.
bq.
bq.
bq. Thanks,
bq.
bq. Mikhail
bq.
bq.
ROWCOL bloom filter not used if multiple columns within same family are
requested in a Get
--
Key: HBASE-2794
URL: https://issues.apache.org/jira/browse/HBASE-2794
Project: HBase
Issue Type: Improvement
Components: performance
Reporter: Kannan Muthukkaruppan
Fix For: 0.92.0
Noticed the following snippet in StoreFile.java:Scanner:shouldSeek():
{code}
switch(bloomFilterType) {
case ROW:
key = row;
break;
case ROWCOL:
if (columns.size() == 1) {
byte[] col = columns.first();
key = Bytes.add(row, col);
break;
}
//$FALL-THROUGH$
default:
return true;
}
{code}
If columns.size 1, then we currently don't take advantage of the bloom
filter. We should optimize this to check bloom for each of columns and if
none of the columns are present in the bloom avoid opening the file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
[jira] [Commented] (HBASE-2794) ROWCOL bloom filter not used if multiple columns within same family are requested in a Get
[
https://issues.apache.org/jira/browse/HBASE-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117523#comment-13117523
]
jirapos...@reviews.apache.org commented on HBASE-2794:
--
bq. On 2011-09-28 17:42:46, Ted Yu wrote:
bq. This is an important feature.
bq.
bq. Since the boolean parameter, forward, correlates so closely with reseek,
can we give it a better name ?
bq. I was thinking about either reseek or forwardOnly.
We have a few diffs in the pipeline that depend on this one. Can we rename the
boolean flag after we commit those diffs?
- Mikhail
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2084/#review2137
---
On 2011-09-28 16:03:52, Mikhail Bautin wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2084/
bq. ---
bq.
bq. (Updated 2011-09-28 16:03:52)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Previously we only used row-column Bloom filters for scans that only
requested one column. We have seen production queries that request up to 200
columns, and with say ~6 store files per store (region / column family
combination) this might have resulted in 1200 block read operations in the
worst case. With this diff we will be avoiding seeks on store files that we
know don't contain the row/column of interest when using an
ExplicitColumnTracker. The performance should remain the same for column range
queries.
bq.
bq.
bq. This addresses bug HBASE-2794.
bq. https://issues.apache.org/jira/browse/HBASE-2794
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
08d3ba4
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
ac2348e
bq.src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 4aa72de
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
68cdac5
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
fd9e7ef
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
9d9895c
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
6cdada7
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7cbdb98
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/KeyValue.java 585c4a8
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
f5173c4
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java a3d778e
bq.src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java
32f88fb
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueHeap.java
a5d13f7
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java
baee696
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/2084/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Existing unit tests. A new unit test (TestScanWithBloomError). Load
testing using HBaseTest.
bq.
bq.
bq. Thanks,
bq.
bq. Mikhail
bq.
bq.
ROWCOL bloom filter not used if multiple columns within same family are
requested in a Get
--
Key: HBASE-2794
URL: https://issues.apache.org/jira/browse/HBASE-2794
Project: HBase
Issue Type: Improvement
Components: performance
Reporter: Kannan Muthukkaruppan
Fix For: 0.92.0
Noticed the following snippet in StoreFile.java:Scanner:shouldSeek():
{code}
switch(bloomFilterType) {
case ROW:
key = row;
break;
case ROWCOL:
if (columns.size() == 1) {
byte[] col = columns.first();
key = Bytes.add(row, col);
break;
}
//$FALL-THROUGH$
default:
return true;
}
{code}
If columns.size 1, then we currently don't take advantage of the bloom
filter. We should optimize this to check bloom for each of columns and if
none of the columns are present in the bloom avoid opening the file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
[jira] [Commented] (HBASE-2794) ROWCOL bloom filter not used if multiple columns within same family are requested in a Get
[
https://issues.apache.org/jira/browse/HBASE-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117524#comment-13117524
]
jirapos...@reviews.apache.org commented on HBASE-2794:
--
bq. On 2011-09-28 17:42:46, Ted Yu wrote:
bq. This is an important feature.
bq.
bq. Since the boolean parameter, forward, correlates so closely with reseek,
can we give it a better name ?
bq. I was thinking about either reseek or forwardOnly.
bq.
bq. Mikhail Bautin wrote:
bq. We have a few diffs in the pipeline that depend on this one. Can we
rename the boolean flag after we commit those diffs?
I am fine with the current name of forward.
- Ted
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2084/#review2137
---
On 2011-09-28 16:03:52, Mikhail Bautin wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2084/
bq. ---
bq.
bq. (Updated 2011-09-28 16:03:52)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Previously we only used row-column Bloom filters for scans that only
requested one column. We have seen production queries that request up to 200
columns, and with say ~6 store files per store (region / column family
combination) this might have resulted in 1200 block read operations in the
worst case. With this diff we will be avoiding seeks on store files that we
know don't contain the row/column of interest when using an
ExplicitColumnTracker. The performance should remain the same for column range
queries.
bq.
bq.
bq. This addresses bug HBASE-2794.
bq. https://issues.apache.org/jira/browse/HBASE-2794
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
08d3ba4
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
ac2348e
bq.src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 4aa72de
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
68cdac5
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
fd9e7ef
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
9d9895c
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
6cdada7
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7cbdb98
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/KeyValue.java 585c4a8
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
f5173c4
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java a3d778e
bq.src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java
32f88fb
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueHeap.java
a5d13f7
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java
baee696
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/2084/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Existing unit tests. A new unit test (TestScanWithBloomError). Load
testing using HBaseTest.
bq.
bq.
bq. Thanks,
bq.
bq. Mikhail
bq.
bq.
ROWCOL bloom filter not used if multiple columns within same family are
requested in a Get
--
Key: HBASE-2794
URL: https://issues.apache.org/jira/browse/HBASE-2794
Project: HBase
Issue Type: Improvement
Components: performance
Reporter: Kannan Muthukkaruppan
Fix For: 0.92.0
Noticed the following snippet in StoreFile.java:Scanner:shouldSeek():
{code}
switch(bloomFilterType) {
case ROW:
key = row;
break;
case ROWCOL:
if (columns.size() == 1) {
byte[] col = columns.first();
key = Bytes.add(row, col);
break;
}
//$FALL-THROUGH$
default:
return true;
}
{code}
If columns.size 1, then we currently don't take advantage of the bloom
filter. We should optimize this to check bloom for each of columns and if
none of the columns are present in the bloom avoid opening the file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact
[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class
[ https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117546#comment-13117546 ] jirapos...@reviews.apache.org commented on HBASE-4422: -- bq. On 2011-09-29 00:50:58, Andrew Purtell wrote: bq. /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java, line 112 bq. https://reviews.apache.org/r/2089/diff/1/?file=46305#file46305line112 bq. bq. CacheConfig could extend Configuration? It's only extra constants and constructors, really. bq. bq. Jonathan Gray wrote: bq. I considered doing that. But they seem orthogonal in most of their usage and I think it'd be weird to pass our Configuration all the way down into HFile reading and such. Changing config values dynamically (like disabling caching for a specific read) would then require changing the base Configuration or would the local booleans in the CacheConfig not match what's in the underlying Configuration? Fair enough, but it's a common pattern to mutate a Configuration object and then pass it along. - Andrew --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2089/#review2145 --- On 2011-09-28 19:56:14, Jonathan Gray wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2089/ bq. --- bq. bq. (Updated 2011-09-28 19:56:14) bq. bq. bq. Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Creates a new CacheConfig class and moves almost everything block cache related into this single class. Adding new configuration params and booleans and such should be much better. bq. bq. All tests are NOT passing yet, still working on it, but wanted to have something up today. Basically code complete but broken :) bq. bq. bq. This addresses bug HBASE-4422. bq. https://issues.apache.org/jira/browse/HBASE-4422 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java PRE-CREATION bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 1177030 bq. /src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 1177030 bq. /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1177030 bq.
[jira] [Updated] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog
[ https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HBASE-4487: Attachment: appendNoSync6.txt All unit tests pass now (expect TestDistributedLogSplitting, TestRollingRestart, TestHTablePool), but I am seeing the same test to fail on trunk, so these failures do not seem to be related to this patch. The one reference to System.err.println() is a printUsage() message that is needed only if u want to run the unit test as a standalone command line utility. There is a single test TestIncrement that creates a 100 threads and ensures that all the concurrent increments match the final expected result. There is a benchmark TestHLogBench that measures the performance of the appendNoSync call. The increment operation can release the rowlock before sync-ing the Hlog Key: HBASE-4487 URL: https://issues.apache.org/jira/browse/HBASE-4487 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: appendNoSync4.txt, appendNoSync5.txt, appendNoSync6.txt This allows for better throughput when there are hot rows.I have seen this change make a single row update improve from 400 increments/sec/server to 4000 increments/sec/server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog
[ https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117556#comment-13117556 ] jirapos...@reviews.apache.org commented on HBASE-4487: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2116/ --- Review request for hbase and Ted Yu. Summary --- The increment operation releases the rowlock before doing the sync to the HLog. This improves performance of increments on hot rows. Introduced method HLog.appendNoSync() that returns a txid. The increment method then release the rowlock and invokes HLog.sync(txid). The HLog.sync(txid) returns only if all the transactions upto the one identified by that txid has been successfully sycned to HDFS. There is a single test TestIncrement that creates a 100 threads and ensures that all the concurrent increments match the final expected result. There is a benchmark TestHLogBench that measures the performance of the appendNoSync call. This addresses bug HBASE-4487. https://issues.apache.org/jira/browse/HBASE-4487 Diffs - /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1177401 /src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 1177401 /src/test/java/org/apache/hadoop/hbase/regionserver/TestIncrement.java PRE-CREATION /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogBench.java PRE-CREATION /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java 1177401 /src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java 1177401 Diff: https://reviews.apache.org/r/2116/diff Testing --- All unit tests pass now (expect TestDistributedLogSplitting, TestRollingRestart, TestHTablePool), but I am seeing the same test to fail on trunk, so these failures do not seem to be related to this patch. Thanks, Dhruba The increment operation can release the rowlock before sync-ing the Hlog Key: HBASE-4487 URL: https://issues.apache.org/jira/browse/HBASE-4487 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: appendNoSync4.txt, appendNoSync5.txt, appendNoSync6.txt This allows for better throughput when there are hot rows.I have seen this change make a single row update improve from 400 increments/sec/server to 4000 increments/sec/server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog
[ https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117557#comment-13117557 ] dhruba borthakur commented on HBASE-4487: - Review board: https://reviews.apache.org/r/2116/ The increment operation can release the rowlock before sync-ing the Hlog Key: HBASE-4487 URL: https://issues.apache.org/jira/browse/HBASE-4487 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: appendNoSync4.txt, appendNoSync5.txt, appendNoSync6.txt This allows for better throughput when there are hot rows.I have seen this change make a single row update improve from 400 increments/sec/server to 4000 increments/sec/server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store metadata into the transaction log
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117562#comment-13117562 ] dhruba borthakur commented on HBASE-4477: - I'm curious if passing the cluster ID to internalPut and internalDelete is necessary? I do not think that this is necessary. We can clean them up when we merge all the parameters to the coprocessor api. Ability for an application to store metadata into the transaction log - Key: HBASE-4477 URL: https://issues.apache.org/jira/browse/HBASE-4477 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: coprocessorPut1.txt, coprocessorPut2.txt, hlogMetadata1.txt mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE. The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated HBASE-4070: - Attachment: rs-status-web-ui.jpg Regionserver Web UI, showing loaded coprocessors. [Coprocessors] Improve region server metrics to report loaded coprocessors to master Key: HBASE-4070 URL: https://issues.apache.org/jira/browse/HBASE-4070 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Mingjie Lai Assignee: Eugene Koontz Attachments: HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg HBASE-3512 is about listing loaded cp classes at shell. To make it more generic, we need a way to report this piece of information from region to master (or just at region server level). So later on, we can display the loaded class names at shell as well as web console. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated HBASE-4070: - Attachment: master-web-ui.jpg HBase Master Web UI, showing master and regionserver coprocessors. [Coprocessors] Improve region server metrics to report loaded coprocessors to master Key: HBASE-4070 URL: https://issues.apache.org/jira/browse/HBASE-4070 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Mingjie Lai Assignee: Eugene Koontz Attachments: HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg HBASE-3512 is about listing loaded cp classes at shell. To make it more generic, we need a way to report this piece of information from region to master (or just at region server level). So later on, we can display the loaded class names at shell as well as web console. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated HBASE-4070: - Attachment: (was: HBASE-4070.patch) [Coprocessors] Improve region server metrics to report loaded coprocessors to master Key: HBASE-4070 URL: https://issues.apache.org/jira/browse/HBASE-4070 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Mingjie Lai Assignee: Eugene Koontz Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg HBASE-3512 is about listing loaded cp classes at shell. To make it more generic, we need a way to report this piece of information from region to master (or just at region server level). So later on, we can display the loaded class names at shell as well as web console. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated HBASE-4070: - Attachment: HBASE-4070.patch Display both master as well as regionserver coprocessors in master Web UI. [Coprocessors] Improve region server metrics to report loaded coprocessors to master Key: HBASE-4070 URL: https://issues.apache.org/jira/browse/HBASE-4070 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Mingjie Lai Assignee: Eugene Koontz Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg HBASE-3512 is about listing loaded cp classes at shell. To make it more generic, we need a way to report this piece of information from region to master (or just at region server level). So later on, we can display the loaded class names at shell as well as web console. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3512) Coprocessors: Shell support for listing currently loaded coprocessor set
[ https://issues.apache.org/jira/browse/HBASE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated HBASE-3512: - Status: Patch Available (was: Open) Submitting two patches: HBASE-3125.patch : this is a superset of the patch on HBASE-4070. It is against apache trunk as of now. HBASE-3125-only.patch: this patch is on top of HBASE-4070's patch, showing only the differences between HBASE-4070 and HBASE-3125. Coprocessors: Shell support for listing currently loaded coprocessor set Key: HBASE-3512 URL: https://issues.apache.org/jira/browse/HBASE-3512 Project: HBase Issue Type: Improvement Components: coprocessors Reporter: Andrew Purtell Assignee: Eugene Koontz Fix For: 0.92.0 Attachments: HBASE-3512-only.patch, HBASE-3512.patch Add support to the shell for listing the coprocessors loaded globally on the regionserver and those loaded on a per-table basis. Perhaps by extending the 'status' command. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3512) Coprocessors: Shell support for listing currently loaded coprocessor set
[ https://issues.apache.org/jira/browse/HBASE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117579#comment-13117579 ] Eugene Koontz commented on HBASE-3512: -- Please excuse the typos in the above. HBASE-3512, not HBASE-3125. Coprocessors: Shell support for listing currently loaded coprocessor set Key: HBASE-3512 URL: https://issues.apache.org/jira/browse/HBASE-3512 Project: HBase Issue Type: Improvement Components: coprocessors Reporter: Andrew Purtell Assignee: Eugene Koontz Fix For: 0.92.0 Attachments: HBASE-3512-only.patch, HBASE-3512.patch Add support to the shell for listing the coprocessors loaded globally on the regionserver and those loaded on a per-table basis. Perhaps by extending the 'status' command. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117583#comment-13117583 ] jirapos...@reviews.apache.org commented on HBASE-4070: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2029/ --- (Updated 2011-09-29 20:15:12.832998) Review request for hbase and Mingjie Lai. Changes --- show both tests. Summary --- Proposed fix for HBASE-4070. This addresses bug HBASE-4070. https://issues.apache.org/jira/browse/HBASE-4070 Diffs - src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon abeb850 src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon be6fceb src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java a55a4b1 src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java dbae4fd src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java a069400 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 270f3f3 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b src/main/ruby/hbase/admin.rb b244ffe src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java a8f2a9c Diff: https://reviews.apache.org/r/2029/diff Testing (updated) --- Two new tests : testRegionServerCoprocessorReported() and testMasterServerCoprocessorsReported() added to src/test/java/o.a.h.h/coprocessor/TestCoprocessorEndpoint.java. Thanks, Eugene [Coprocessors] Improve region server metrics to report loaded coprocessors to master Key: HBASE-4070 URL: https://issues.apache.org/jira/browse/HBASE-4070 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Mingjie Lai Assignee: Eugene Koontz Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg HBASE-3512 is about listing loaded cp classes at shell. To make it more generic, we need a way to report this piece of information from region to master (or just at region server level). So later on, we can display the loaded class names at shell as well as web console. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117581#comment-13117581 ] jirapos...@reviews.apache.org commented on HBASE-4070: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2029/ --- (Updated 2011-09-29 20:13:32.064908) Review request for hbase and Mingjie Lai. Changes --- Remove relies on HBASE-4014: this is no longer true. Summary (updated) --- Proposed fix for HBASE-4070. This addresses bug HBASE-4070. https://issues.apache.org/jira/browse/HBASE-4070 Diffs - src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon abeb850 src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon be6fceb src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java a55a4b1 src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java dbae4fd src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java a069400 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 270f3f3 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b src/main/ruby/hbase/admin.rb b244ffe src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java a8f2a9c Diff: https://reviews.apache.org/r/2029/diff Testing --- One new test : testServerManagerCoprocessorReport, added to src/test/java/o.a.h.h/coprocessor/TestCoprocessorEndpoint.java. Thanks, Eugene [Coprocessors] Improve region server metrics to report loaded coprocessors to master Key: HBASE-4070 URL: https://issues.apache.org/jira/browse/HBASE-4070 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Mingjie Lai Assignee: Eugene Koontz Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg HBASE-3512 is about listing loaded cp classes at shell. To make it more generic, we need a way to report this piece of information from region to master (or just at region server level). So later on, we can display the loaded class names at shell as well as web console. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3512) Coprocessors: Shell support for listing currently loaded coprocessor set
[ https://issues.apache.org/jira/browse/HBASE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated HBASE-3512: - Attachment: hbase-shell-session.txt hbase shell session, showing new coprocessor information. Coprocessors: Shell support for listing currently loaded coprocessor set Key: HBASE-3512 URL: https://issues.apache.org/jira/browse/HBASE-3512 Project: HBase Issue Type: Improvement Components: coprocessors Reporter: Andrew Purtell Assignee: Eugene Koontz Fix For: 0.92.0 Attachments: HBASE-3512-only.patch, HBASE-3512.patch, hbase-shell-session.txt Add support to the shell for listing the coprocessors loaded globally on the regionserver and those loaded on a per-table basis. Perhaps by extending the 'status' command. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4418) Show all the hbase configuration in the web ui
[ https://issues.apache.org/jira/browse/HBASE-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117573#comment-13117573 ] Liyin Tang commented on HBASE-4418: --- @stack,is it true that all the patches for hbase trunk should rebase on hadoop trunk or hadoop-0.24 (the latest release) ? Show all the hbase configuration in the web ui -- Key: HBASE-4418 URL: https://issues.apache.org/jira/browse/HBASE-4418 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang The motivation is to show ALL the HBase configuration, which takes effect in the run time, in a global place. So we can easily know which configuration takes effect and what the value is. The configuration shows all the HBase and DFS configuration entry in the configuration file and also includes all the HBase default setting in the code, which is not the config file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-3512) Coprocessors: Shell support for listing currently loaded coprocessor set
[ https://issues.apache.org/jira/browse/HBASE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz reassigned HBASE-3512: Assignee: Eugene Koontz (was: Mingjie Lai) Coprocessors: Shell support for listing currently loaded coprocessor set Key: HBASE-3512 URL: https://issues.apache.org/jira/browse/HBASE-3512 Project: HBase Issue Type: Improvement Components: coprocessors Reporter: Andrew Purtell Assignee: Eugene Koontz Fix For: 0.92.0 Add support to the shell for listing the coprocessors loaded globally on the regionserver and those loaded on a per-table basis. Perhaps by extending the 'status' command. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117574#comment-13117574 ] jirapos...@reviews.apache.org commented on HBASE-4070: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2029/ --- (Updated 2011-09-29 19:56:54.298361) Review request for hbase and Mingjie Lai. Changes --- Updated patch that shows both master as well as regionserver loaded coprocessors. Master UI shows both master and regionserver coprocessor information, while regionserver UI (/rs-status) shows regionserver coprocessor information. See JIRA for screenshots. Summary --- Proposed fix for HBASE-4070. Relies on functionality provided by HBASE-4014. This addresses bug HBASE-4070. https://issues.apache.org/jira/browse/HBASE-4070 Diffs (updated) - src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon abeb850 src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon be6fceb src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java a55a4b1 src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java dbae4fd src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java a069400 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 270f3f3 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b src/main/ruby/hbase/admin.rb b244ffe src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java a8f2a9c Diff: https://reviews.apache.org/r/2029/diff Testing --- One new test : testServerManagerCoprocessorReport, added to src/test/java/o.a.h.h/coprocessor/TestCoprocessorEndpoint.java. Thanks, Eugene [Coprocessors] Improve region server metrics to report loaded coprocessors to master Key: HBASE-4070 URL: https://issues.apache.org/jira/browse/HBASE-4070 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Mingjie Lai Assignee: Eugene Koontz Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg HBASE-3512 is about listing loaded cp classes at shell. To make it more generic, we need a way to report this piece of information from region to master (or just at region server level). So later on, we can display the loaded class names at shell as well as web console. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3512) Coprocessors: Shell support for listing currently loaded coprocessor set
[ https://issues.apache.org/jira/browse/HBASE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated HBASE-3512: - Attachment: HBASE-3512-only.patch HBASE-3512.patch Submitting two patches: HBASE-3125.patch : this is a superset of the patch on HBASE-4070. It is against apache trunk as of now. HBASE-3125-only.patch: this patch is on top of HBASE-4070's patch, showing only the differences between HBASE-4070 and HBASE-3125. Coprocessors: Shell support for listing currently loaded coprocessor set Key: HBASE-3512 URL: https://issues.apache.org/jira/browse/HBASE-3512 Project: HBase Issue Type: Improvement Components: coprocessors Reporter: Andrew Purtell Assignee: Eugene Koontz Fix For: 0.92.0 Attachments: HBASE-3512-only.patch, HBASE-3512.patch Add support to the shell for listing the coprocessors loaded globally on the regionserver and those loaded on a per-table basis. Perhaps by extending the 'status' command. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4499) [replication] Source shouldn't update ZK if it didn't progress
[ https://issues.apache.org/jira/browse/HBASE-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117601#comment-13117601 ] Chris Trezzo commented on HBASE-4499: - @J-D Here is a simple sketch patch. I have not tested it yet. Chris [replication] Source shouldn't update ZK if it didn't progress -- Key: HBASE-4499 URL: https://issues.apache.org/jira/browse/HBASE-4499 Project: HBase Issue Type: Improvement Affects Versions: 0.90.4 Reporter: Jean-Daniel Cryans Priority: Minor Fix For: 0.92.0 A relatively minor optimization to be done in ReplicationSource, currently it calls ReplicationSourceManager.logPositionAndCleanOldLogs whether it made progress or not, generating more load on ZK than necessary. The last position should be kept around so that we can compare. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated HBASE-4070: - Attachment: HBASE-4070.patch Display both master as well as regionserver coprocessors in master Web UI. [Coprocessors] Improve region server metrics to report loaded coprocessors to master Key: HBASE-4070 URL: https://issues.apache.org/jira/browse/HBASE-4070 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Mingjie Lai Assignee: Eugene Koontz Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg HBASE-3512 is about listing loaded cp classes at shell. To make it more generic, we need a way to report this piece of information from region to master (or just at region server level). So later on, we can display the loaded class names at shell as well as web console. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4499) [replication] Source shouldn't update ZK if it didn't progress
[ https://issues.apache.org/jira/browse/HBASE-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Trezzo updated HBASE-4499: Attachment: 4499.txt [replication] Source shouldn't update ZK if it didn't progress -- Key: HBASE-4499 URL: https://issues.apache.org/jira/browse/HBASE-4499 Project: HBase Issue Type: Improvement Affects Versions: 0.90.4 Reporter: Jean-Daniel Cryans Priority: Minor Fix For: 0.92.0 Attachments: 4499.txt A relatively minor optimization to be done in ReplicationSource, currently it calls ReplicationSourceManager.logPositionAndCleanOldLogs whether it made progress or not, generating more load on ZK than necessary. The last position should be kept around so that we can compare. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4513) NOTICES.txt refers to Facebook for Thrift
[
https://issues.apache.org/jira/browse/HBASE-4513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117604#comment-13117604
]
Hudson commented on HBASE-4513:
---
Integrated in HBase-TRUNK #2269 (See
[https://builds.apache.org/job/HBase-TRUNK/2269/])
HBASE-4513 NOTICES.txt refers to Facebook for Thrift
stack :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/NOTICE.txt
NOTICES.txt refers to Facebook for Thrift
-
Key: HBASE-4513
URL: https://issues.apache.org/jira/browse/HBASE-4513
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Joe Pallas
Assignee: stack
Priority: Minor
Fix For: 0.92.0
The NOTICES.txt file says:
{quote}
In addition, this product includes software developed by:
Facebook, Inc. (http://developers.facebook.com/thrift/ -- Page includes the
Thrift Software License)
{quote}
Since Thrift is now an Apache project, this seems to be obsolete.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4512) JVMClusterUtil throwing wrong exception when master thread cannot be created
[
https://issues.apache.org/jira/browse/HBASE-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117603#comment-13117603
]
Hudson commented on HBASE-4512:
---
Integrated in HBase-TRUNK #2269 (See
[https://builds.apache.org/job/HBase-TRUNK/2269/])
HBASE-4512 JVMClusterUtil throwing wrong exception when master thread
cannot be created (Ram) - Did not update the CHANGES.txt
HBASE-4512 JVMClusterUtil throwing wrong exception when master thread cannot be
created (Ram)
ramkrishna :
Files :
* /hbase/trunk/CHANGES.txt
ramkrishna :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/JVMClusterUtil.java
JVMClusterUtil throwing wrong exception when master thread cannot be created
Key: HBASE-4512
URL: https://issues.apache.org/jira/browse/HBASE-4512
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Trivial
Fix For: 0.92.0, 0.94.0, 0.90.5
Attachments: HBASE-4512_trunk.patch, HBASE_4253_0.90.patch
In JVMClusterUtil.MasterThread createMasterThread
{code}
throw new RuntimeException(Failed construction of RegionServer: +
{code}
It must be failed construction of Master.
Very trivial one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4514) Change the co-processor API to take in a single argument object
Change the co-processor API to take in a single argument object --- Key: HBASE-4514 URL: https://issues.apache.org/jira/browse/HBASE-4514 Project: HBase Issue Type: Improvement Components: coprocessors Reporter: dhruba borthakur The current coprocessor RegionObserver API takes in a bunch of arguments into each of the method calls. Instead, it will be better if we encapsulate these arguments into a single object. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4145) Provide metrics for hbase client
[ https://issues.apache.org/jira/browse/HBASE-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117608#comment-13117608 ] jirapos...@reviews.apache.org commented on HBASE-4145: -- bq. On 2011-09-29 13:33:05, ramkrishna vasudevan wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java, line 2 bq. https://reviews.apache.org/r/1674/diff/4/?file=46377#file46377line2 bq. bq. I think we need not give copyright information. Fixed. - Ming --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1674/#review2150 --- On 2011-09-29 21:00:18, Ming Ma wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1674/ bq. --- bq. bq. (Updated 2011-09-29 21:00:18) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. 1. Collect client-side scan related metrics during scan operation. It is turned off by default. bq. 2. TableInputFormat enables metrics collection on scan and pass the data to mapreduce framework. It only works with new mapreduce APIs that allow TableInputFormat to get access to mapreduce Counter. bq. 3. Clean up some minor issues in tableInputFormat as well as test code. bq. bq. bq. This addresses bug hbase-4145. bq. https://issues.apache.org/jira/browse/hbase-4145 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReader.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java 1176942 bq. bq. Diff: https://reviews.apache.org/r/1674/diff bq. bq. bq. Testing bq. --- bq. bq. 1. Verified on a small cluster. bq. 2. Existing unit tests. bq. 3. Added new tests. bq. bq. bq. Thanks, bq. bq. Ming bq. bq. Provide metrics for hbase client Key: HBASE-4145 URL: https://issues.apache.org/jira/browse/HBASE-4145 Project: HBase Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma Attachments: HBaseClientSideMetrics.jpg Sometimes it is useful to get some metrics from hbase client point of view. This will help understand the metrics for scan/TableInputFormat map job scenario. What to capture, for example, for each ResultScanner object, 1. The number of RPC calls to RSs. 2. The delta time between consecutive RPC calls in the current serialized scan implementation. 3. The number of RPC retry to RSs. 4. The number of NotServingRegionException got. 5. The number of remote RPC calls. This excludes those call that hbase client calls the RS on the same machine. 6. The number of regions accessed. How to capture 1. Metrics framework works for a fixed number of metrics. It doesn't fit this scenario. 2. Use some TBD solution in HBase to capture such dynamic metrics. If we assume there is a solution in HBase that HBase client can use to log such kind of metrics, TableInputFormat can pass in mapreduce task ID as application scan ID to HBase client as small addition to existing scan API; and HBase client can log metrics accordingly with such ID. That will allow query, analysis later on the metrics data for specific map reduce job. 3. Expose via MapReduce counter. It lacks certain features, for example, there is no good way to access the metrics on per map instance; the
[jira] [Resolved] (HBASE-4477) Ability for an application to store metadata into the transaction log
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Gray resolved HBASE-4477. -- Resolution: Fixed Fix Version/s: 0.92.0 Hadoop Flags: Reviewed Committed to trunk and 92 branch. Thanks for all the reviews and feedback all, nice stuff Dhruba. Dhruba opened HBASE-4514 for the follow-up discussed here. Ability for an application to store metadata into the transaction log - Key: HBASE-4477 URL: https://issues.apache.org/jira/browse/HBASE-4477 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.92.0 Attachments: coprocessorPut1.txt, coprocessorPut2.txt, hlogMetadata1.txt mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE. The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4145) Provide metrics for hbase client
[ https://issues.apache.org/jira/browse/HBASE-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117611#comment-13117611 ] jirapos...@reviews.apache.org commented on HBASE-4145: -- bq. On 2011-09-29 03:48:24, Ted Yu wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java, line 43 bq. https://reviews.apache.org/r/1674/diff/3/?file=46268#file46268line43 bq. bq. Should read 'can be easily' Fixed. - Ming --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1674/#review2148 --- On 2011-09-29 21:00:18, Ming Ma wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1674/ bq. --- bq. bq. (Updated 2011-09-29 21:00:18) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. 1. Collect client-side scan related metrics during scan operation. It is turned off by default. bq. 2. TableInputFormat enables metrics collection on scan and pass the data to mapreduce framework. It only works with new mapreduce APIs that allow TableInputFormat to get access to mapreduce Counter. bq. 3. Clean up some minor issues in tableInputFormat as well as test code. bq. bq. bq. This addresses bug hbase-4145. bq. https://issues.apache.org/jira/browse/hbase-4145 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReader.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java 1176942 bq. bq. Diff: https://reviews.apache.org/r/1674/diff bq. bq. bq. Testing bq. --- bq. bq. 1. Verified on a small cluster. bq. 2. Existing unit tests. bq. 3. Added new tests. bq. bq. bq. Thanks, bq. bq. Ming bq. bq. Provide metrics for hbase client Key: HBASE-4145 URL: https://issues.apache.org/jira/browse/HBASE-4145 Project: HBase Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma Attachments: HBaseClientSideMetrics.jpg Sometimes it is useful to get some metrics from hbase client point of view. This will help understand the metrics for scan/TableInputFormat map job scenario. What to capture, for example, for each ResultScanner object, 1. The number of RPC calls to RSs. 2. The delta time between consecutive RPC calls in the current serialized scan implementation. 3. The number of RPC retry to RSs. 4. The number of NotServingRegionException got. 5. The number of remote RPC calls. This excludes those call that hbase client calls the RS on the same machine. 6. The number of regions accessed. How to capture 1. Metrics framework works for a fixed number of metrics. It doesn't fit this scenario. 2. Use some TBD solution in HBase to capture such dynamic metrics. If we assume there is a solution in HBase that HBase client can use to log such kind of metrics, TableInputFormat can pass in mapreduce task ID as application scan ID to HBase client as small addition to existing scan API; and HBase client can log metrics accordingly with such ID. That will allow query, analysis later on the metrics data for specific map reduce job. 3. Expose via MapReduce counter. It lacks certain features, for example, there is no good way to access the metrics on per map instance; the MapReduce framework only performs
[jira] [Commented] (HBASE-4145) Provide metrics for hbase client
[
https://issues.apache.org/jira/browse/HBASE-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117609#comment-13117609
]
jirapos...@reviews.apache.org commented on HBASE-4145:
--
bq. On 2011-09-29 04:18:54, Ted Yu wrote:
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java,
line 48
bq. https://reviews.apache.org/r/1674/diff/4/?file=46377#file46377line48
bq.
bq. Should be declared as implementing VersionedWritable.
The issue with VersionedWritable is it throws VersionMismatchException if the
version doesn't match.
public void readFields(DataInput in) throws IOException {
byte version = in.readByte(); // read version
if (version != getVersion())
throw new VersionMismatchException(getVersion(), version);
}
I want to make it backward compatible to support version = getVersion(). The
program could catch VersionMismatchException, however, there is no way to find
out the expectedVersion and foundVersion, given they are private members.
public class VersionMismatchException extends IOException {
private byte expectedVersion;
private byte foundVersion;
...
}
Any other suggestions, or Is it something that need to be fixed in
VersionedWritable, VersionMismatchException?
bq. On 2011-09-29 04:18:54, Ted Yu wrote:
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java,
line 76
bq. https://reviews.apache.org/r/1674/diff/4/?file=46377#file46377line76
bq.
bq. It is a bit hard to read this counter.
Fixed.
bq. On 2011-09-29 04:18:54, Ted Yu wrote:
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java,
line 94
bq. https://reviews.apache.org/r/1674/diff/4/?file=46377#file46377line94
bq.
bq. This is count of regions scanned, right ?
bq. If so, please name it that way.
Todd suggested to rename it from COUNT_OF_REGIONS to REGIONS, given the fact
that it is a counter is implicit in mapreduce framework.
bq. On 2011-09-29 04:18:54, Ted Yu wrote:
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java,
line 127
bq. https://reviews.apache.org/r/1674/diff/4/?file=46377#file46377line127
bq.
bq. mb should be included in the exception.
Fixed.
bq. On 2011-09-29 04:18:54, Ted Yu wrote:
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java,
line 133
bq. https://reviews.apache.org/r/1674/diff/4/?file=46377#file46377line133
bq.
bq. Why do we need this check again ?
Fixed.
bq. On 2011-09-29 04:18:54, Ted Yu wrote:
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java,
line 143
bq. https://reviews.apache.org/r/1674/diff/4/?file=46377#file46377line143
bq.
bq. Value of version should be included here.
Fixed.
bq. On 2011-09-29 04:18:54, Ted Yu wrote:
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java,
line 151
bq. https://reviews.apache.org/r/1674/diff/4/?file=46377#file46377line151
bq.
bq. I think we should have else block where the unsupported mb is logged.
Fixed.
bq. On 2011-09-29 04:18:54, Ted Yu wrote:
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java,
line 52
bq. https://reviews.apache.org/r/1674/diff/4/?file=46380#file46380line52
bq.
bq. This name doesn't really match the constant above. I think HBase
mapreduce Counters would be better.
The name should show up in mapreduce UI and report. Other group names don't
have mapreduce. So keep it as HBase Counters and rename the variable.
bq. On 2011-09-29 04:18:54, Ted Yu wrote:
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java,
line 83
bq. https://reviews.apache.org/r/1674/diff/4/?file=46380#file46380line83
bq.
bq. This should not be a tongue twister.
bq. How about naming it retrieveGetCounterWithStrings ?
Fixed.
bq. On 2011-09-29 04:18:54, Ted Yu wrote:
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java,
line 232
bq. https://reviews.apache.org/r/1674/diff/4/?file=46380#file46380line232
bq.
bq. Shall we create the Object array outside the for loop and only fill
in Metric name here ?
Fixed. Don't create Object at all, pass the parameters directly.
- Ming
---
This is an automatically generated e-mail. To reply, visit:
[jira] [Commented] (HBASE-4145) Provide metrics for hbase client
[ https://issues.apache.org/jira/browse/HBASE-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117612#comment-13117612 ] jirapos...@reviews.apache.org commented on HBASE-4145: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1674/ --- (Updated 2011-09-29 21:00:18.525989) Review request for hbase. Changes --- Thanks for the review, Ted, Ram. Most are fixed. Please find comments inline. Summary --- 1. Collect client-side scan related metrics during scan operation. It is turned off by default. 2. TableInputFormat enables metrics collection on scan and pass the data to mapreduce framework. It only works with new mapreduce APIs that allow TableInputFormat to get access to mapreduce Counter. 3. Clean up some minor issues in tableInputFormat as well as test code. This addresses bug hbase-4145. https://issues.apache.org/jira/browse/hbase-4145 Diffs (updated) - http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReader.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java 1176942 Diff: https://reviews.apache.org/r/1674/diff Testing --- 1. Verified on a small cluster. 2. Existing unit tests. 3. Added new tests. Thanks, Ming Provide metrics for hbase client Key: HBASE-4145 URL: https://issues.apache.org/jira/browse/HBASE-4145 Project: HBase Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma Attachments: HBaseClientSideMetrics.jpg Sometimes it is useful to get some metrics from hbase client point of view. This will help understand the metrics for scan/TableInputFormat map job scenario. What to capture, for example, for each ResultScanner object, 1. The number of RPC calls to RSs. 2. The delta time between consecutive RPC calls in the current serialized scan implementation. 3. The number of RPC retry to RSs. 4. The number of NotServingRegionException got. 5. The number of remote RPC calls. This excludes those call that hbase client calls the RS on the same machine. 6. The number of regions accessed. How to capture 1. Metrics framework works for a fixed number of metrics. It doesn't fit this scenario. 2. Use some TBD solution in HBase to capture such dynamic metrics. If we assume there is a solution in HBase that HBase client can use to log such kind of metrics, TableInputFormat can pass in mapreduce task ID as application scan ID to HBase client as small addition to existing scan API; and HBase client can log metrics accordingly with such ID. That will allow query, analysis later on the metrics data for specific map reduce job. 3. Expose via MapReduce counter. It lacks certain features, for example, there is no good way to access the metrics on per map instance; the MapReduce framework only performs sum on the counter values so it is tricky to find the max of certain metrics in all mapper instances. However, it might be good enough for now. With this approach, the metrics value will be available via MapReduce counter. a) Have ResultScanner return a new ResultScannerMetrics interface. b) TableInputFormat will access data from ResultScannerMetrics and populate MapReduce counters accordingly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2794) ROWCOL bloom filter not used if multiple columns within same family are requested in a Get
[
https://issues.apache.org/jira/browse/HBASE-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117615#comment-13117615
]
jirapos...@reviews.apache.org commented on HBASE-2794:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2084/
---
(Updated 2011-09-29 21:05:20.334849)
Review request for hbase.
Changes
---
Addressing Jonathan's comments.
Summary
---
Previously we only used row-column Bloom filters for scans that only requested
one column. We have seen production queries that request up to 200 columns, and
with say ~6 store files per store (region / column family combination) this
might have resulted in 1200 block read operations in the worst case. With this
diff we will be avoiding seeks on store files that we know don't contain the
row/column of interest when using an ExplicitColumnTracker. The performance
should remain the same for column range queries.
This addresses bug HBASE-2794.
https://issues.apache.org/jira/browse/HBASE-2794
Diffs (updated)
-
src/main/java/org/apache/hadoop/hbase/KeyValue.java 585c4a8
src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
f5173c4
src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java a3d778e
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7cbdb98
src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java 9d9895c
src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
6cdada7
src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 4aa72de
src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
68cdac5
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java fd9e7ef
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
08d3ba4
src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java ac2348e
src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java
32f88fb
src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueHeap.java
a5d13f7
src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java
baee696
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java
PRE-CREATION
Diff: https://reviews.apache.org/r/2084/diff
Testing
---
Existing unit tests. A new unit test (TestScanWithBloomError). Load testing
using HBaseTest.
Thanks,
Mikhail
ROWCOL bloom filter not used if multiple columns within same family are
requested in a Get
--
Key: HBASE-2794
URL: https://issues.apache.org/jira/browse/HBASE-2794
Project: HBase
Issue Type: Improvement
Components: performance
Reporter: Kannan Muthukkaruppan
Fix For: 0.92.0
Noticed the following snippet in StoreFile.java:Scanner:shouldSeek():
{code}
switch(bloomFilterType) {
case ROW:
key = row;
break;
case ROWCOL:
if (columns.size() == 1) {
byte[] col = columns.first();
key = Bytes.add(row, col);
break;
}
//$FALL-THROUGH$
default:
return true;
}
{code}
If columns.size 1, then we currently don't take advantage of the bloom
filter. We should optimize this to check bloom for each of columns and if
none of the columns are present in the bloom avoid opening the file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4062) Multi-column scanner unit test
[ https://issues.apache.org/jira/browse/HBASE-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bautin resolved HBASE-4062. --- Resolution: Fixed Fix Version/s: 0.94.0 TestMultiColumnScanner has been committed to trunk. Multi-column scanner unit test -- Key: HBASE-4062 URL: https://issues.apache.org/jira/browse/HBASE-4062 Project: HBase Issue Type: Improvement Components: regionserver, test Affects Versions: 0.94.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Labels: test Fix For: 0.94.0 Attachments: test-multi-column-scanner.patch Adding a unit test for the multi-column scanner. We are using this this to test an optimization we are making to multi-column scans using row-column Bloom filters. The scanner creates multiple StoreFiles for a single column family, each containing a randomized set of columns with different timestamps, and then tests scanning through the whole region with all possible sets of columns specified in the query. Point deletes (deletes of a specific timestamp) are also tested. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4497) If region opening fails after updating META HBCK reports it as inconsistent and scanning the region throws NSRE
[ https://issues.apache.org/jira/browse/HBASE-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117617#comment-13117617 ] jirapos...@reviews.apache.org commented on HBASE-4497: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2118/ --- Review request for hbase. Summary --- Adds a checkAndPut that takes a timestamp This addresses bug hbase-4497. https://issues.apache.org/jira/browse/hbase-4497 Diffs - src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02 src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7cbdb98 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 0c06f4f src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 99b34cc Diff: https://reviews.apache.org/r/2118/diff Testing --- Thanks, Michael If region opening fails after updating META HBCK reports it as inconsistent and scanning the region throws NSRE --- Key: HBASE-4497 URL: https://issues.apache.org/jira/browse/HBASE-4497 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Priority: Critical As per the discussion in the mail chain HBCK reporting of possible mismatch in RS assignment this JIRA is created. Consider two RS- RS1 and RS2. A region tries to open in RS1. But it takes a while. The RS1 has still not updated meta and transitioned the node from OPENING to OPENED So timeout assigns the region to RS2. RS2 successfully updates the META and opens the region. Now RS1 tries to act on the region by first updating the META and then transiting the node to OPENING to OPENED. RS1 transiting the node to OPENING to OPENED will fail. But the META entry will have RS1 as the latest. Now HBCK reports this as an inconsistency and if we try to scan the Region we get NotServingRegionException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID
[ https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117618#comment-13117618 ] jirapos...@reviews.apache.org commented on HBASE-4507: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2118/ --- (Updated 2011-09-29 21:10:29.425308) Review request for hbase. Changes --- Fixed Summary. Summary (updated) --- Adds a checkAndPut that takes a timestamp This addresses bug hbase-4507. https://issues.apache.org/jira/browse/hbase-4507 Diffs - src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02 src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7cbdb98 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 0c06f4f src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 99b34cc Diff: https://reviews.apache.org/r/2118/diff Testing --- Thanks, Michael Create checkAndPut variant that exposes timestamp / UUID Key: HBASE-4507 URL: https://issues.apache.org/jira/browse/HBASE-4507 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Michael checked the checkAndPut which doesn't expose timestamp. A variant of checkAndPut should be created to expose timestamp which is written into a column specified by additional parameters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID
[ https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4507: - Status: Patch Available (was: Open) I posted first cut at a patch that adds the timestamp. Also has guards against an update going in with same timestamp as the update its overwriting, the case Ming outlines at tail of hbase-4497. Create checkAndPut variant that exposes timestamp / UUID Key: HBASE-4507 URL: https://issues.apache.org/jira/browse/HBASE-4507 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Michael checked the checkAndPut which doesn't expose timestamp. A variant of checkAndPut should be created to expose timestamp which is written into a column specified by additional parameters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID
[ https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack reassigned HBASE-4507: Assignee: stack Create checkAndPut variant that exposes timestamp / UUID Key: HBASE-4507 URL: https://issues.apache.org/jira/browse/HBASE-4507 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: stack Michael checked the checkAndPut which doesn't expose timestamp. A variant of checkAndPut should be created to expose timestamp which is written into a column specified by additional parameters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4145) Provide metrics for hbase client
[ https://issues.apache.org/jira/browse/HBASE-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117621#comment-13117621 ] jirapos...@reviews.apache.org commented on HBASE-4145: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1674/#review2175 --- Only one minor comment left, see below. If test suite passes, I would vote +1 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java https://reviews.apache.org/r/1674/#comment5068 VersionedWritable from hadoop left something to be desired. Such discussion came up during HBASE-2195. We can address this elsewhere. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java https://reviews.apache.org/r/1674/#comment5069 I meant that this counter can be named REGIONS_SCANNED because its value may be lower than the total number of regions in the table(s). - Ted On 2011-09-29 21:00:18, Ming Ma wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1674/ bq. --- bq. bq. (Updated 2011-09-29 21:00:18) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. 1. Collect client-side scan related metrics during scan operation. It is turned off by default. bq. 2. TableInputFormat enables metrics collection on scan and pass the data to mapreduce framework. It only works with new mapreduce APIs that allow TableInputFormat to get access to mapreduce Counter. bq. 3. Clean up some minor issues in tableInputFormat as well as test code. bq. bq. bq. This addresses bug hbase-4145. bq. https://issues.apache.org/jira/browse/hbase-4145 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReader.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java 1176942 bq. bq. Diff: https://reviews.apache.org/r/1674/diff bq. bq. bq. Testing bq. --- bq. bq. 1. Verified on a small cluster. bq. 2. Existing unit tests. bq. 3. Added new tests. bq. bq. bq. Thanks, bq. bq. Ming bq. bq. Provide metrics for hbase client Key: HBASE-4145 URL: https://issues.apache.org/jira/browse/HBASE-4145 Project: HBase Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma Attachments: HBaseClientSideMetrics.jpg Sometimes it is useful to get some metrics from hbase client point of view. This will help understand the metrics for scan/TableInputFormat map job scenario. What to capture, for example, for each ResultScanner object, 1. The number of RPC calls to RSs. 2. The delta time between consecutive RPC calls in the current serialized scan implementation. 3. The number of RPC retry to RSs. 4. The number of NotServingRegionException got. 5. The number of remote RPC calls. This excludes those call that hbase client calls the RS on the same machine. 6. The number of regions accessed. How to capture 1. Metrics framework works for a fixed number of metrics. It doesn't fit this scenario. 2. Use some TBD solution in HBase to capture such dynamic metrics. If we assume there is a solution in HBase that HBase client can use to log such kind of metrics, TableInputFormat can pass in mapreduce task ID as application scan
[jira] [Assigned] (HBASE-4515) User.getCurrent() can cause NPE.
[
https://issues.apache.org/jira/browse/HBASE-4515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh reassigned HBASE-4515:
-
Assignee: Jonathan Hsieh
User.getCurrent() can cause NPE.
Key: HBASE-4515
URL: https://issues.apache.org/jira/browse/HBASE-4515
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
When testing with miniclusters that shutdown and are restarted, sometimes a
call to User.getCurrent().getName() NPEs when attempting to restart hbase.
Oddly this happens consistently on particular branches and not on others. I
don't know or understand why this happens but it has something to do with the
getCurrentUGI call in o.a.h.h.security.User.HadoopUser sometimes returning
null and sometimes returning data.
{code}
private HadoopUser() {
try {
ugi = (UserGroupInformation) callStatic(getCurrentUGI);
if (ugi == null) {
LOG.warn(Although successfully retrieved UserGroupInformation
+ it was null!);
}
} catch (RuntimeException re) {
{code}
This patch essentially is a workaround -- it propagates the null so that
clients can check and avoid the NPE.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4515) User.getCurrent() can cause NPE.
User.getCurrent() can cause NPE.
Key: HBASE-4515
URL: https://issues.apache.org/jira/browse/HBASE-4515
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Hsieh
When testing with miniclusters that shutdown and are restarted, sometimes a
call to User.getCurrent().getName() NPEs when attempting to restart hbase.
Oddly this happens consistently on particular branches and not on others. I
don't know or understand why this happens but it has something to do with the
getCurrentUGI call in o.a.h.h.security.User.HadoopUser sometimes returning
null and sometimes returning data.
{code}
private HadoopUser() {
try {
ugi = (UserGroupInformation) callStatic(getCurrentUGI);
if (ugi == null) {
LOG.warn(Although successfully retrieved UserGroupInformation
+ it was null!);
}
} catch (RuntimeException re) {
{code}
This patch essentially is a workaround -- it propagates the null so that
clients can check and avoid the NPE.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4515) User.getCurrent() can cause NPE.
[
https://issues.apache.org/jira/browse/HBASE-4515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117623#comment-13117623
]
Jonathan Hsieh commented on HBASE-4515:
---
A stack trace of the error that I'd like to avoid.
{code}
2011-09-29 11:38:45,823 ERROR [Thread-341] hbase.MiniHBaseCluster(201): Error
starting cluster
java.lang.NullPointerException
at org.apache.hadoop.hbase.security.User.getName(User.java:71)
at
org.apache.hadoop.hbase.HBaseTestingUtility.getDifferentUser(HBaseTestingUtility.java:1421)
at
org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:191)
at
org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:76)
at
org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:61)
at
org.apache.hadoop.hbase.HBaseTestingUtility.restartHBaseCluster(HBaseTestingUtility.java:505)
at
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuild.testMetaRebuild(TestOfflineMetaRebuild.java:306)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
{code}
User.getCurrent() can cause NPE.
Key: HBASE-4515
URL: https://issues.apache.org/jira/browse/HBASE-4515
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
When testing with miniclusters that shutdown and are restarted, sometimes a
call to User.getCurrent().getName() NPEs when attempting to restart hbase.
Oddly this happens consistently on particular branches and not on others. I
don't know or understand why this happens but it has something to do with the
getCurrentUGI call in o.a.h.h.security.User.HadoopUser sometimes returning
null and sometimes returning data.
{code}
private HadoopUser() {
try {
ugi = (UserGroupInformation) callStatic(getCurrentUGI);
if (ugi == null) {
LOG.warn(Although successfully retrieved UserGroupInformation
+ it was null!);
}
} catch (RuntimeException re) {
{code}
This patch essentially is a workaround -- it propagates the null so that
clients can check and avoid the NPE.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4503) Purge deprecated HBaseClusterTestCase
[ https://issues.apache.org/jira/browse/HBASE-4503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117627#comment-13117627 ] jirapos...@reviews.apache.org commented on HBASE-4503: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2119/ --- Review request for hbase. Summary --- Replace deprecated HBaseClusterTestCase and subclasses. This addresses bug hbase-4503. https://issues.apache.org/jira/browse/hbase-4503 Diffs - Diff: https://reviews.apache.org/r/2119/diff Testing --- Unit tests pass Thanks, Michael Purge deprecated HBaseClusterTestCase - Key: HBASE-4503 URL: https://issues.apache.org/jira/browse/HBASE-4503 Project: HBase Issue Type: Improvement Reporter: stack Assignee: stack Attachments: 4503-v2.txt, 4503.txt It could gain us a few minutes on overall test run in the cases where we don't spin up a cluster for each test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
