date:20110930

2011-09-30 Thread ramkrishna.s.vasudevan (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117970#comment-13117970
 ] 

Ted Yu commented on HBASE-4357:
---

I think points 1 and 2 above are reasonable.

 Region in transition - in closing state
 ---

 Key: HBASE-4357
 URL: https://issues.apache.org/jira/browse/HBASE-4357
 Project: HBase
  Issue Type: Bug
Reporter: Ming Ma
Assignee: Ming Ma

 Got the following during testing, 
 1. On a given machine, kill RS process id. Then kill HMaster process id.
 2. Start RS first via bin/hbase-daemon.sh --config ./conf start 
 regionserver.. Start HMaster via bin/hbase-daemon.sh --config ./conf start 
 master.
 One region of a table stayed in closing state.
 According to zookeeper,
 794a6ff17a4de0dd0a19b984ba18eea9 
 miweng_500region,H\xB49X\x10bM\xB1,1315338786464.794a6ff17a4de0dd0a19b984ba18eea9.
  state=CLOSING, ts=Wed Sep 07 17:21:44 PDT 2011 (75701s ago), 
 server=sea-esxi-0,6,1315428682281 
 According to .META. table, the region has been assigned to from sea-esxi-0 to 
 sea-esxi-4.
 miweng_500region,H\xB49X\x10bM\xB1,1315338786464.794a6ff17a4de0dd0a19b984ba18eea9.
  sea-esxi-4:60030  H\xB49X\x10bM\xB1 I7K\xC6\xA7\xEF\x9D\x90 0 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog

2011-09-30 Thread Ted Yu (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4487:
--

Fix Version/s: 0.94.0
 Hadoop Flags: Incompatible change

 The increment operation can release the rowlock before sync-ing the Hlog
 

 Key: HBASE-4487
 URL: https://issues.apache.org/jira/browse/HBASE-4487
 Project: HBase
  Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: appendNoSync4.txt, appendNoSync5.txt, appendNoSync6.txt


 This allows for better throughput when there are hot rows.I have seen this 
 change make a single row update improve from 400 increments/sec/server to 
 4000 increments/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4511) There is data loss when master failovers


[ 
https://issues.apache.org/jira/browse/HBASE-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117978#comment-13117978
 ] 

ramkrishna.s.vasudevan commented on HBASE-4511:
---

@Gao
I think generally the split of log will happen only if the RS dies and the RS 
node gets deleted.
If that does not happen then the split logs of that crashing RS may not happen.
May be in actual scenario though the master does not process as dead server by 
splitting the logs it will any way wait for ServerShutDownHandler to process it.

what do you feel Gao?

 There is data loss when master failovers
 

 Key: HBASE-4511
 URL: https://issues.apache.org/jira/browse/HBASE-4511
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.0
Reporter: gaojinchao
Priority: Critical
 Fix For: 0.92.0

 Attachments: 
 org.apache.hadoop.hbase.master.TestMasterFailover-output.rar


 It goes like this:
 Master crashed ,  at the same time RS with meta is crashing, but RS doesn't 
 eixt.
 Master startups again and finds all living RS. 
 Master verifies the meta failed,  because this RS is crashing.
 Master reassigns the meta, but it doesn't split the Hlog. 
 So some meta data is loss.
 About the logs of a failover test case fail. 
 //It said that we want to kill a RS
 2011-09-28 19:54:45,694 INFO  [Thread-988] regionserver.HRegionServer(1443): 
 STOPPED: Killing for unit test
 2011-09-28 19:54:45,694 INFO  [Thread-988] master.TestMasterFailover(1007): 
 RS 192.168.2.102,54385,1317264874629 killed 
 //Rs didn't crash. 
 2011-09-28 19:54:51,763 INFO  [Master:0;192.168.2.102,54557,1317264885720] 
 master.HMaster(458): Registering server found up in zk: 
 192.168.2.102,54385,1317264874629
 2011-09-28 19:54:51,763 INFO  [Master:0;192.168.2.102,54557,1317264885720] 
 master.ServerManager(232): Registering 
 server=192.168.2.102,54385,1317264874629
 2011-09-28 19:54:51,770 DEBUG [Master:0;192.168.2.102,54557,1317264885720] 
 zookeeper.ZKUtil(491): master:54557-0x132b31adbb30005 Unable to get data of 
 znode /hbase/unassigned/1028785192 because node does not exist (not an error)
 2011-09-28 19:54:51,771 DEBUG [Master:0;192.168.2.102,54557,1317264885720] 
 zookeeper.ZKUtil(1003): master:54557-0x132b31adbb30005 Retrieved 33 byte(s) 
 of data from znode /hbase/root-region-server and set watcher; 
 192.168.2.102,54383,131726487...
 //Meta verification failed and ressigned the meta. So all the regions in the 
 meta is loss.
 2011-09-28 19:54:51,773 INFO  [Master:0;192.168.2.102,54557,1317264885720] 
 catalog.CatalogTracker(476): Failed verification of .META.,,1 at 
 address=192.168.2.102,54385,1317264874629; 
 org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: 
 org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server 
 192.168.2.102,54385,1317264874629 not running, aborting
 2011-09-28 19:54:51,773 DEBUG [Master:0;192.168.2.102,54557,1317264885720] 
 catalog.CatalogTracker(316): new .META. server: 
 192.168.2.102,54385,1317264874629 isn't valid. Cached .META. server: null
 2011-09-28 19:54:52,274 DEBUG [Master:0;192.168.2.102,54557,1317264885720] 
 zookeeper.ZKUtil(1003): master:54557-0x132b31adbb30005 Retrieved 33 byte(s) 
 of data from znode /hbase/root-region-server and set watcher; 
 192.168.2.102,54383,131726487...
 2011-09-28 19:54:52,277 INFO  [Master:0;192.168.2.102,54557,1317264885720] 
 catalog.CatalogTracker(476): Failed verification of .META.,,1 at 
 address=192.168.2.102,54385,1317264874629; 
 org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: 
 org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server 
 192.168.2.102,54385,1317264874629 not running, aborting
 2011-09-28 19:54:52,277 DEBUG [Master:0;192.168.2.102,54557,1317264885720] 
 catalog.CatalogTracker(316): new .META. server: 
 192.168.2.102,54385,1317264874629 isn't valid. Cached .META. server: null
 2011-09-28 19:54:52,778 DEBUG [Master:0;192.168.2.102,54557,1317264885720] 
 zookeeper.ZKUtil(1003): master:54557-0x132b31adbb30005 Retrieved 33 byte(s) 
 of data from znode /hbase/root-region-server and set watcher; 
 192.168.2.102,54383,131726487...
 2011-09-28 19:54:52,782 INFO  [Master:0;192.168.2.102,54557,1317264885720] 
 catalog.CatalogTracker(476): Failed verification of .META.,,1 at 
 address=192.168.2.102,54385,1317264874629; 
 org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: 
 org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server 
 192.168.2.102,54385,1317264874629 not running, aborting
 2011-09-28 19:54:52,782 DEBUG [Master:0;192.168.2.102,54557,1317264885720] 
 catalog.CatalogTracker(316): new .META. server: 
 192.168.2.102,54385,1317264874629 isn't valid. Cached .META. server: null
 2011-09-28

[jira] [Commented] (HBASE-4212) TestMasterFailover fails occasionally

2011-09-30 Thread ramkrishna.s.vasudevan (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117979#comment-13117979
 ] 

ramkrishna.s.vasudevan commented on HBASE-4212:
---

And at the same time even if the check is not there i do not think there will 
be a problem. Correct me if am wrong.

 TestMasterFailover fails occasionally
 -

 Key: HBASE-4212
 URL: https://issues.apache.org/jira/browse/HBASE-4212
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.4
Reporter: gaojinchao
Assignee: gaojinchao
 Fix For: 0.90.5

 Attachments: HBASE-4212_TrunkV1.patch, HBASE-4212_branch90V1.patch


 It seems a bug. The root in RIT can't be moved..
 In the failover process, it enforces root on-line. But not clean zk node. 
 test will wait forever.
   void processFailover() throws KeeperException, IOException, 
 InterruptedException {
  
 // we enforce on-line root.
 HServerInfo hsi =
   
 this.serverManager.getHServerInfo(this.catalogTracker.getMetaLocation());
 regionOnline(HRegionInfo.FIRST_META_REGIONINFO, hsi);
 hsi = 
 this.serverManager.getHServerInfo(this.catalogTracker.getRootLocation());
 regionOnline(HRegionInfo.ROOT_REGIONINFO, hsi);
 It seems that we should wait finished as meta region 
   int assignRootAndMeta()
   throws InterruptedException, IOException, KeeperException {
 int assigned = 0;
 long timeout = this.conf.getLong(hbase.catalog.verification.timeout, 
 1000);
 // Work on ROOT region.  Is it in zk in transition?
 boolean rit = this.assignmentManager.
   
 processRegionInTransitionAndBlockUntilAssigned(HRegionInfo.ROOT_REGIONINFO);
 if (!catalogTracker.verifyRootRegionLocation(timeout)) {
   this.assignmentManager.assignRoot();
   this.catalogTracker.waitForRoot();
   //we need add this code and guarantee that the transition has completed
   this.assignmentManager.waitForAssignment(HRegionInfo.ROOT_REGIONINFO);
   assigned++;
 }
 logs:
 2011-08-16 07:45:40,715 DEBUG 
 [RegionServer:0;C4S2.site,47710,1313495126115-EventThread] 
 zookeeper.ZooKeeperWatcher(252): regionserver:47710-0x131d2690f780004 
 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, 
 path=/hbase/unassigned/70236052
 2011-08-16 07:45:40,715 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0] 
 zookeeper.ZKAssign(712): regionserver:47710-0x131d2690f780004 Successfully 
 transitioned node 70236052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
 2011-08-16 07:45:40,715 DEBUG [Thread-760-EventThread] 
 zookeeper.ZooKeeperWatcher(252): master:60701-0x131d2690f780009 Received 
 ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, 
 path=/hbase/unassigned/70236052
 2011-08-16 07:45:40,716 INFO  [PostOpenDeployTasks:70236052] 
 catalog.RootLocationEditor(62): Setting ROOT region location in ZooKeeper as 
 C4S2.site:47710
 2011-08-16 07:45:40,716 DEBUG [Thread-760-EventThread] 
 zookeeper.ZKUtil(1109): master:60701-0x131d2690f780009 Retrieved 52 byte(s) 
 of data from znode /hbase/unassigned/70236052 and set watcher; 
 region=-ROOT-,,0, server=C4S2.site,47710,1313495126115, 
 state=RS_ZK_REGION_OPENING
 2011-08-16 07:45:40,717 DEBUG [Thread-760-EventThread] 
 master.AssignmentManager(477): Handling transition=RS_ZK_REGION_OPENING, 
 server=C4S2.site,47710,1313495126115, region=70236052/-ROOT-
 2011-08-16 07:45:40,725 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0] 
 zookeeper.ZKAssign(661): regionserver:47710-0x131d2690f780004 Attempting to 
 transition node 70236052/-ROOT- from RS_ZK_REGION_OPENING to 
 RS_ZK_REGION_OPENED
 2011-08-16 07:45:40,727 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0] 
 zookeeper.ZKUtil(1109): regionserver:47710-0x131d2690f780004 Retrieved 52 
 byte(s) of data from znode /hbase/unassigned/70236052; data=region=-ROOT-,,0, 
 server=C4S2.site,47710,1313495126115, state=RS_ZK_REGION_OPENING
 2011-08-16 07:45:40,740 DEBUG 
 [RegionServer:0;C4S2.site,47710,1313495126115-EventThread] 
 zookeeper.ZooKeeperWatcher(252): regionserver:47710-0x131d2690f780004 
 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, 
 path=/hbase/unassigned/70236052
 2011-08-16 07:45:40,740 DEBUG [Thread-760-EventThread] 
 zookeeper.ZooKeeperWatcher(252): master:60701-0x131d2690f780009 Received 
 ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, 
 path=/hbase/unassigned/70236052
 2011-08-16 07:45:40,740 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0] 
 zookeeper.ZKAssign(712): regionserver:47710-0x131d2690f780004 Successfully 
 transitioned node 70236052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
 2011-08-16 07:45:40,741 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0] 
 handler.OpenRegionHandler(121): Opened

[jira] [Commented] (HBASE-4212) TestMasterFailover fails occasionally


[ 
https://issues.apache.org/jira/browse/HBASE-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118041#comment-13118041
 ] 

Hudson commented on HBASE-4212:
---

Integrated in HBase-0.92 #33 (See 
[https://builds.apache.org/job/HBase-0.92/33/])
HBASE-4212  TestMasterFailover fails occasionally (Gao Jinchao)

tedyu : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java


 TestMasterFailover fails occasionally
 -

 Key: HBASE-4212
 URL: https://issues.apache.org/jira/browse/HBASE-4212
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.4
Reporter: gaojinchao
Assignee: gaojinchao
 Fix For: 0.90.5

 Attachments: HBASE-4212_TrunkV1.patch, HBASE-4212_branch90V1.patch


 It seems a bug. The root in RIT can't be moved..
 In the failover process, it enforces root on-line. But not clean zk node. 
 test will wait forever.
   void processFailover() throws KeeperException, IOException, 
 InterruptedException {
  
 // we enforce on-line root.
 HServerInfo hsi =
   
 this.serverManager.getHServerInfo(this.catalogTracker.getMetaLocation());
 regionOnline(HRegionInfo.FIRST_META_REGIONINFO, hsi);
 hsi = 
 this.serverManager.getHServerInfo(this.catalogTracker.getRootLocation());
 regionOnline(HRegionInfo.ROOT_REGIONINFO, hsi);
 It seems that we should wait finished as meta region 
   int assignRootAndMeta()
   throws InterruptedException, IOException, KeeperException {
 int assigned = 0;
 long timeout = this.conf.getLong(hbase.catalog.verification.timeout, 
 1000);
 // Work on ROOT region.  Is it in zk in transition?
 boolean rit = this.assignmentManager.
   
 processRegionInTransitionAndBlockUntilAssigned(HRegionInfo.ROOT_REGIONINFO);
 if (!catalogTracker.verifyRootRegionLocation(timeout)) {
   this.assignmentManager.assignRoot();
   this.catalogTracker.waitForRoot();
   //we need add this code and guarantee that the transition has completed
   this.assignmentManager.waitForAssignment(HRegionInfo.ROOT_REGIONINFO);
   assigned++;
 }
 logs:
 2011-08-16 07:45:40,715 DEBUG 
 [RegionServer:0;C4S2.site,47710,1313495126115-EventThread] 
 zookeeper.ZooKeeperWatcher(252): regionserver:47710-0x131d2690f780004 
 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, 
 path=/hbase/unassigned/70236052
 2011-08-16 07:45:40,715 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0] 
 zookeeper.ZKAssign(712): regionserver:47710-0x131d2690f780004 Successfully 
 transitioned node 70236052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
 2011-08-16 07:45:40,715 DEBUG [Thread-760-EventThread] 
 zookeeper.ZooKeeperWatcher(252): master:60701-0x131d2690f780009 Received 
 ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, 
 path=/hbase/unassigned/70236052
 2011-08-16 07:45:40,716 INFO  [PostOpenDeployTasks:70236052] 
 catalog.RootLocationEditor(62): Setting ROOT region location in ZooKeeper as 
 C4S2.site:47710
 2011-08-16 07:45:40,716 DEBUG [Thread-760-EventThread] 
 zookeeper.ZKUtil(1109): master:60701-0x131d2690f780009 Retrieved 52 byte(s) 
 of data from znode /hbase/unassigned/70236052 and set watcher; 
 region=-ROOT-,,0, server=C4S2.site,47710,1313495126115, 
 state=RS_ZK_REGION_OPENING
 2011-08-16 07:45:40,717 DEBUG [Thread-760-EventThread] 
 master.AssignmentManager(477): Handling transition=RS_ZK_REGION_OPENING, 
 server=C4S2.site,47710,1313495126115, region=70236052/-ROOT-
 2011-08-16 07:45:40,725 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0] 
 zookeeper.ZKAssign(661): regionserver:47710-0x131d2690f780004 Attempting to 
 transition node 70236052/-ROOT- from RS_ZK_REGION_OPENING to 
 RS_ZK_REGION_OPENED
 2011-08-16 07:45:40,727 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0] 
 zookeeper.ZKUtil(1109): regionserver:47710-0x131d2690f780004 Retrieved 52 
 byte(s) of data from znode /hbase/unassigned/70236052; data=region=-ROOT-,,0, 
 server=C4S2.site,47710,1313495126115, state=RS_ZK_REGION_OPENING
 2011-08-16 07:45:40,740 DEBUG 
 [RegionServer:0;C4S2.site,47710,1313495126115-EventThread] 
 zookeeper.ZooKeeperWatcher(252): regionserver:47710-0x131d2690f780004 
 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, 
 path=/hbase/unassigned/70236052
 2011-08-16 07:45:40,740 DEBUG [Thread-760-EventThread] 
 zookeeper.ZooKeeperWatcher(252): master:60701-0x131d2690f780009 Received 
 ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, 
 path=/hbase/unassigned/70236052
 2011-08-16 07:45:40,740 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0] 
 zookeeper.ZKAssign(712): regionserver:47710-0x131d2690f780004 Successfully 
 transitioned node 70236052 from RS_ZK_REGION_OPENING to

[jira] [Commented] (HBASE-3130) [replication] ReplicationSource can't recover from session expired on remote clusters


[ 
https://issues.apache.org/jira/browse/HBASE-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118042#comment-13118042
 ] 

Hudson commented on HBASE-3130:
---

Integrated in HBase-0.92 #33 (See 
[https://builds.apache.org/job/HBase-0.92/33/])
HBASE-3130  [replication] ReplicationSource can't recover from session
   expired on remote clusters (Chris Trezzo via JD)
   HBASE-4499  [replication] Source shouldn't update ZK if it didn't progress
   (Chris Trezzo via JD)

jdcryans : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeer.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationPeer.java


 [replication] ReplicationSource can't recover from session expired on remote 
 clusters
 -

 Key: HBASE-3130
 URL: https://issues.apache.org/jira/browse/HBASE-3130
 Project: HBase
  Issue Type: Bug
  Components: replication
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Chris Trezzo
 Fix For: 0.92.0

 Attachments: 3130-v2.txt, 3130-v3.txt, 3130-v4.txt, 3130.txt, 
 HBASE-3130-v5.patch


 Currently ReplicationSource cannot recover when its zookeeper connection to 
 its remote cluster expires. HLogs are still being tracked, but a cluster 
 restart is required to continue replication (or a rolling restart).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4412) No need to retry scan operation on the same server in case of RegionServerStoppedException


[ 
https://issues.apache.org/jira/browse/HBASE-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118040#comment-13118040
 ] 

Hudson commented on HBASE-4412:
---

Integrated in HBase-0.92 #33 (See 
[https://builds.apache.org/job/HBase-0.92/33/])
HBASE-4412  No need to retry scan operation on the same server in case of
   RegionServerStoppedException (Ming Ma)

tedyu : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java


 No need to retry scan operation on the same server in case of 
 RegionServerStoppedException
 --

 Key: HBASE-4412
 URL: https://issues.apache.org/jira/browse/HBASE-4412
 Project: HBase
  Issue Type: Bug
Reporter: Ming Ma
Assignee: Ming Ma
 Fix For: 0.92.0

 Attachments: HBASE-4412-trunk.patch


 ScannerCallable.java doesn't retry on the same server in case of 
 NotServingRegionException.
 if (ioe instanceof NotServingRegionException) {
   // Throw a DNRE so that we break out of cycle of calling NSRE
   // when what we need is to open scanner against new location.
   // Attach NSRE to signal client that it needs to resetup scanner.
   throw new DoNotRetryIOException(Reset scanner, ioe);
 In the scenario when region server is in shutdown process, 
 RegionServerStoppedException will be thrown. ScannerCallable still retry. It 
 isn't necessary.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4499) [replication] Source shouldn't update ZK if it didn't progress


[ 
https://issues.apache.org/jira/browse/HBASE-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118039#comment-13118039
 ] 

Hudson commented on HBASE-4499:
---

Integrated in HBase-0.92 #33 (See 
[https://builds.apache.org/job/HBase-0.92/33/])
HBASE-3130  [replication] ReplicationSource can't recover from session
   expired on remote clusters (Chris Trezzo via JD)
   HBASE-4499  [replication] Source shouldn't update ZK if it didn't progress
   (Chris Trezzo via JD)

jdcryans : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeer.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationPeer.java


 [replication] Source shouldn't update ZK if it didn't progress
 --

 Key: HBASE-4499
 URL: https://issues.apache.org/jira/browse/HBASE-4499
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Assignee: Chris Trezzo
Priority: Minor
 Fix For: 0.92.0

 Attachments: 4499.txt


 A relatively minor optimization to be done in ReplicationSource, currently it 
 calls ReplicationSourceManager.logPositionAndCleanOldLogs whether it made 
 progress or not, generating more load on ZK than necessary. The last position 
 should be kept around so that we can compare.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4145) Provide metrics for hbase client

[
https://issues.apache.org/jira/browse/HBASE-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118134#comment-13118134
]

Hudson commented on HBASE-4145:
---

Integrated in HBase-TRUNK #2272 (See
[https://builds.apache.org/job/HBase-TRUNK/2272/])
HBASE-4145 Provide metrics for hbase client, add ScanMetrics.java
HBASE-4145 Provide metrics for hbase client (Ming Ma)

tedyu :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java

tedyu :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReader.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
*
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
*
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java

Provide metrics for hbase client

Key: HBASE-4145
URL: https://issues.apache.org/jira/browse/HBASE-4145
Project: HBase
Issue Type: Improvement
Reporter: Ming Ma
Assignee: Ming Ma
Fix For: 0.94.0

Attachments: HBaseClientSideMetrics.jpg

Sometimes it is useful to get some metrics from hbase client point of view.
This will help understand the metrics for scan/TableInputFormat map job
scenario.
What to capture, for example, for each ResultScanner object,
1. The number of RPC calls to RSs.
2. The delta time between consecutive RPC calls in the current serialized
scan implementation.
3. The number of RPC retry to RSs.
4. The number of NotServingRegionException got.
5. The number of remote RPC calls. This excludes those call that hbase client
calls the RS on the same machine.
6. The number of regions accessed.
How to capture
1. Metrics framework works for a fixed number of metrics. It doesn't fit this
scenario.
2. Use some TBD solution in HBase to capture such dynamic metrics. If we
assume there is a solution in HBase that HBase client can use to log such
kind of metrics, TableInputFormat can pass in mapreduce task ID as
application scan ID to HBase client as small addition to existing scan API;
and HBase client can log metrics accordingly with such ID. That will allow
query, analysis later on the metrics data for specific map reduce job.
3. Expose via MapReduce counter. It lacks certain features, for example,
there is no good way to access the metrics on per map instance; the MapReduce
framework only performs sum on the counter values so it is tricky to find the
max of certain metrics in all mapper instances. However, it might be good
enough for now. With this approach, the metrics value will be available via
MapReduce counter.
a) Have ResultScanner return a new ResultScannerMetrics interface.
b) TableInputFormat will access data from ResultScannerMetrics and populate
MapReduce counters accordingly.

[jira] [Commented] (HBASE-4499) [replication] Source shouldn't update ZK if it didn't progress


[ 
https://issues.apache.org/jira/browse/HBASE-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118132#comment-13118132
 ] 

Hudson commented on HBASE-4499:
---

Integrated in HBase-TRUNK #2272 (See 
[https://builds.apache.org/job/HBase-TRUNK/2272/])
HBASE-3130  [replication] ReplicationSource can't recover from session
   expired on remote clusters (Chris Trezzo via JD)
   HBASE-4499  [replication] Source shouldn't update ZK if it didn't progress
   (Chris Trezzo via JD)

jdcryans : 
Files : 
* /hbase/trunk/CHANGES.txt
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeer.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationPeer.java


 [replication] Source shouldn't update ZK if it didn't progress
 --

 Key: HBASE-4499
 URL: https://issues.apache.org/jira/browse/HBASE-4499
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Assignee: Chris Trezzo
Priority: Minor
 Fix For: 0.92.0

 Attachments: 4499.txt


 A relatively minor optimization to be done in ReplicationSource, currently it 
 calls ReplicationSourceManager.logPositionAndCleanOldLogs whether it made 
 progress or not, generating more load on ZK than necessary. The last position 
 should be kept around so that we can compare.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4412) No need to retry scan operation on the same server in case of RegionServerStoppedException


[ 
https://issues.apache.org/jira/browse/HBASE-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118133#comment-13118133
 ] 

Hudson commented on HBASE-4412:
---

Integrated in HBase-TRUNK #2272 (See 
[https://builds.apache.org/job/HBase-TRUNK/2272/])
HBASE-4412  No need to retry scan operation on the same server in case of
   RegionServerStoppedException (Ming Ma)

tedyu : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java


 No need to retry scan operation on the same server in case of 
 RegionServerStoppedException
 --

 Key: HBASE-4412
 URL: https://issues.apache.org/jira/browse/HBASE-4412
 Project: HBase
  Issue Type: Bug
Reporter: Ming Ma
Assignee: Ming Ma
 Fix For: 0.92.0

 Attachments: HBASE-4412-trunk.patch


 ScannerCallable.java doesn't retry on the same server in case of 
 NotServingRegionException.
 if (ioe instanceof NotServingRegionException) {
   // Throw a DNRE so that we break out of cycle of calling NSRE
   // when what we need is to open scanner against new location.
   // Attach NSRE to signal client that it needs to resetup scanner.
   throw new DoNotRetryIOException(Reset scanner, ioe);
 In the scenario when region server is in shutdown process, 
 RegionServerStoppedException will be thrown. ScannerCallable still retry. It 
 isn't necessary.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3130) [replication] ReplicationSource can't recover from session expired on remote clusters


[ 
https://issues.apache.org/jira/browse/HBASE-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118136#comment-13118136
 ] 

Hudson commented on HBASE-3130:
---

Integrated in HBase-TRUNK #2272 (See 
[https://builds.apache.org/job/HBase-TRUNK/2272/])
HBASE-3130  [replication] ReplicationSource can't recover from session
   expired on remote clusters (Chris Trezzo via JD)
   HBASE-4499  [replication] Source shouldn't update ZK if it didn't progress
   (Chris Trezzo via JD)

jdcryans : 
Files : 
* /hbase/trunk/CHANGES.txt
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeer.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationPeer.java


 [replication] ReplicationSource can't recover from session expired on remote 
 clusters
 -

 Key: HBASE-3130
 URL: https://issues.apache.org/jira/browse/HBASE-3130
 Project: HBase
  Issue Type: Bug
  Components: replication
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Chris Trezzo
 Fix For: 0.92.0

 Attachments: 3130-v2.txt, 3130-v3.txt, 3130-v4.txt, 3130.txt, 
 HBASE-3130-v5.patch


 Currently ReplicationSource cannot recover when its zookeeper connection to 
 its remote cluster expires. HLogs are still being tracked, but a cluster 
 restart is required to continue replication (or a rolling restart).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4212) TestMasterFailover fails occasionally

2011-09-30 Thread ramkrishna.s.vasudevan (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118135#comment-13118135
 ] 

Hudson commented on HBASE-4212:
---

Integrated in HBase-TRUNK #2272 (See 
[https://builds.apache.org/job/HBase-TRUNK/2272/])
HBASE-4212  TestMasterFailover fails occasionally (Gao Jinchao)

tedyu : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java


 TestMasterFailover fails occasionally
 -

 Key: HBASE-4212
 URL: https://issues.apache.org/jira/browse/HBASE-4212
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.4
Reporter: gaojinchao
Assignee: gaojinchao
 Fix For: 0.90.5

 Attachments: HBASE-4212_TrunkV1.patch, HBASE-4212_branch90V1.patch


 It seems a bug. The root in RIT can't be moved..
 In the failover process, it enforces root on-line. But not clean zk node. 
 test will wait forever.
   void processFailover() throws KeeperException, IOException, 
 InterruptedException {
  
 // we enforce on-line root.
 HServerInfo hsi =
   
 this.serverManager.getHServerInfo(this.catalogTracker.getMetaLocation());
 regionOnline(HRegionInfo.FIRST_META_REGIONINFO, hsi);
 hsi = 
 this.serverManager.getHServerInfo(this.catalogTracker.getRootLocation());
 regionOnline(HRegionInfo.ROOT_REGIONINFO, hsi);
 It seems that we should wait finished as meta region 
   int assignRootAndMeta()
   throws InterruptedException, IOException, KeeperException {
 int assigned = 0;
 long timeout = this.conf.getLong(hbase.catalog.verification.timeout, 
 1000);
 // Work on ROOT region.  Is it in zk in transition?
 boolean rit = this.assignmentManager.
   
 processRegionInTransitionAndBlockUntilAssigned(HRegionInfo.ROOT_REGIONINFO);
 if (!catalogTracker.verifyRootRegionLocation(timeout)) {
   this.assignmentManager.assignRoot();
   this.catalogTracker.waitForRoot();
   //we need add this code and guarantee that the transition has completed
   this.assignmentManager.waitForAssignment(HRegionInfo.ROOT_REGIONINFO);
   assigned++;
 }
 logs:
 2011-08-16 07:45:40,715 DEBUG 
 [RegionServer:0;C4S2.site,47710,1313495126115-EventThread] 
 zookeeper.ZooKeeperWatcher(252): regionserver:47710-0x131d2690f780004 
 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, 
 path=/hbase/unassigned/70236052
 2011-08-16 07:45:40,715 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0] 
 zookeeper.ZKAssign(712): regionserver:47710-0x131d2690f780004 Successfully 
 transitioned node 70236052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
 2011-08-16 07:45:40,715 DEBUG [Thread-760-EventThread] 
 zookeeper.ZooKeeperWatcher(252): master:60701-0x131d2690f780009 Received 
 ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, 
 path=/hbase/unassigned/70236052
 2011-08-16 07:45:40,716 INFO  [PostOpenDeployTasks:70236052] 
 catalog.RootLocationEditor(62): Setting ROOT region location in ZooKeeper as 
 C4S2.site:47710
 2011-08-16 07:45:40,716 DEBUG [Thread-760-EventThread] 
 zookeeper.ZKUtil(1109): master:60701-0x131d2690f780009 Retrieved 52 byte(s) 
 of data from znode /hbase/unassigned/70236052 and set watcher; 
 region=-ROOT-,,0, server=C4S2.site,47710,1313495126115, 
 state=RS_ZK_REGION_OPENING
 2011-08-16 07:45:40,717 DEBUG [Thread-760-EventThread] 
 master.AssignmentManager(477): Handling transition=RS_ZK_REGION_OPENING, 
 server=C4S2.site,47710,1313495126115, region=70236052/-ROOT-
 2011-08-16 07:45:40,725 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0] 
 zookeeper.ZKAssign(661): regionserver:47710-0x131d2690f780004 Attempting to 
 transition node 70236052/-ROOT- from RS_ZK_REGION_OPENING to 
 RS_ZK_REGION_OPENED
 2011-08-16 07:45:40,727 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0] 
 zookeeper.ZKUtil(1109): regionserver:47710-0x131d2690f780004 Retrieved 52 
 byte(s) of data from znode /hbase/unassigned/70236052; data=region=-ROOT-,,0, 
 server=C4S2.site,47710,1313495126115, state=RS_ZK_REGION_OPENING
 2011-08-16 07:45:40,740 DEBUG 
 [RegionServer:0;C4S2.site,47710,1313495126115-EventThread] 
 zookeeper.ZooKeeperWatcher(252): regionserver:47710-0x131d2690f780004 
 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, 
 path=/hbase/unassigned/70236052
 2011-08-16 07:45:40,740 DEBUG [Thread-760-EventThread] 
 zookeeper.ZooKeeperWatcher(252): master:60701-0x131d2690f780009 Received 
 ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, 
 path=/hbase/unassigned/70236052
 2011-08-16 07:45:40,740 DEBUG [RS_OPEN_ROOT-C4S2.site,47710,1313495126115-0] 
 zookeeper.ZKAssign(712): regionserver:47710-0x131d2690f780004 Successfully 
 transitioned node 70236052 from RS_ZK_REGION_OPENING to

[jira] [Assigned] (HBASE-2794) ROWCOL bloom filter not used if multiple columns within same family are requested in a Get

2011-09-30 Thread Ted Yu (Assigned) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu reassigned HBASE-2794:
-

Assignee: Mikhail Bautin

 ROWCOL bloom filter not used if multiple columns within same family are 
 requested in a Get
 --

 Key: HBASE-2794
 URL: https://issues.apache.org/jira/browse/HBASE-2794
 Project: HBase
  Issue Type: Improvement
  Components: performance
Reporter: Kannan Muthukkaruppan
Assignee: Mikhail Bautin
 Fix For: 0.92.0


 Noticed the following snippet in StoreFile.java:Scanner:shouldSeek():
 {code}
 switch(bloomFilterType) {
   case ROW:
 key = row;
 break;
   case ROWCOL:
 if (columns.size() == 1) {
   byte[] col = columns.first();
   key = Bytes.add(row, col);
   break;
 }
 //$FALL-THROUGH$
   default:
 return true;
 }
 {code}
 If columns.size  1, then we currently don't take advantage of the bloom 
 filter.  We should optimize this to check bloom for each of columns and if 
 none of the columns are present in the bloom avoid opening the file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog


[ 
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118177#comment-13118177
 ] 

ramkrishna.s.vasudevan commented on HBASE-4487:
---

{code}
-synchronized (this.updateLock) {
-  if (this.closed) {
-return;
-  }
+syncer(this.unflushedEntries.get()); // sync all pending items
{code}

The patch shows that the above 'if' check is been removed.  So it is needed.  

 The increment operation can release the rowlock before sync-ing the Hlog
 

 Key: HBASE-4487
 URL: https://issues.apache.org/jira/browse/HBASE-4487
 Project: HBase
  Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: appendNoSync4.txt, appendNoSync5.txt, appendNoSync6.txt


 This allows for better throughput when there are hot rows.I have seen this 
 change make a single row update improve from 400 increments/sec/server to 
 4000 increments/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog

2011-09-30 Thread dhruba borthakur (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118183#comment-13118183
 ] 

dhruba borthakur commented on HBASE-4487:
-

Hi Ramakrishna, from my understanding, HLog.syncer() used to acquire the 
updateLock and then check this.closed, then used to *release* the updateLock 
and proceed with the sync. So, essentially the check was providing no 
protection against the concurrent execution of the body of the code in syncer() 
vs some other thread marking this.closed to true.

But I agree with you, that even prior to this patch, there is a rare 
possibility that a region server shutdown (via HLog.close) can race with a 
HLog.sync() that is in progress and cause exceptions. If I can recreate that 
race in a test case, I will submit it as part of another JIRA, does that sound 
reasonable?



 The increment operation can release the rowlock before sync-ing the Hlog
 

 Key: HBASE-4487
 URL: https://issues.apache.org/jira/browse/HBASE-4487
 Project: HBase
  Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: appendNoSync4.txt, appendNoSync5.txt, appendNoSync6.txt


 This allows for better throughput when there are hot rows.I have seen this 
 change make a single row update improve from 400 increments/sec/server to 
 4000 increments/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog

2011-09-30 Thread dhruba borthakur (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

dhruba borthakur updated HBASE-4487:


Release Note: The increment operation releases the rowlock before doing the 
sync to the HLog. This improves performance of increments on hot rows. 

 The increment operation can release the rowlock before sync-ing the Hlog
 

 Key: HBASE-4487
 URL: https://issues.apache.org/jira/browse/HBASE-4487
 Project: HBase
  Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: appendNoSync4.txt, appendNoSync5.txt, appendNoSync6.txt


 This allows for better throughput when there are hot rows.I have seen this 
 change make a single row update improve from 400 increments/sec/server to 
 4000 increments/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog

2011-09-30 Thread dhruba borthakur (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118186#comment-13118186
 ] 

dhruba borthakur commented on HBASE-4487:
-

 I'd mark this an 'incompatible change' rather than an 'improvement' just so 
 its more likely folks

I could not find the place where to mark it as incompatible. Can you pl tell 
me where to mark it as such?

 The increment operation can release the rowlock before sync-ing the Hlog
 

 Key: HBASE-4487
 URL: https://issues.apache.org/jira/browse/HBASE-4487
 Project: HBase
  Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: appendNoSync4.txt, appendNoSync5.txt, appendNoSync6.txt


 This allows for better throughput when there are hot rows.I have seen this 
 change make a single row update improve from 400 increments/sec/server to 
 4000 increments/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog

2011-09-30 Thread Andrew Purtell (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118189#comment-13118189
 ] 

Andrew Purtell commented on HBASE-4487:
---

bq. I could not find the place where to mark it as incompatible. Can you pl 
tell me where to mark it as such?

This is a committer requirement Dhruba, don't worry about it, unless you 
disagree about the characterization. There is a checkbox to mark when resolving 
the issue and where the changelog line goes in CHANGES.txt is different.

 The increment operation can release the rowlock before sync-ing the Hlog
 

 Key: HBASE-4487
 URL: https://issues.apache.org/jira/browse/HBASE-4487
 Project: HBase
  Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: appendNoSync4.txt, appendNoSync5.txt, appendNoSync6.txt


 This allows for better throughput when there are hot rows.I have seen this 
 change make a single row update improve from 400 increments/sec/server to 
 4000 increments/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog

2011-09-30 Thread Andrew Purtell (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118193#comment-13118193
]

Andrew Purtell commented on HBASE-4487:
---

My comment lines up with Stack's on the code review, basically: This is a new
code path for increments, and it means that one cannot guarantee that any
individual increment has been committed to the WAL. For the increment case this
walks back some representations we make about HBase for the sake of
performance. I have mixed feelings about that. This should be configurable.
Happy to agree the default is enabled.

I believe there is another JIRA somewhere about making different config
examples for users depending on their risk tolerance or interest in
performance. Enabling this and deferred flushing etc. makes sense in a
performance profile; conversely turned off in a strict profile.

The increment operation can release the rowlock before sync-ing the Hlog

Key: HBASE-4487
URL: https://issues.apache.org/jira/browse/HBASE-4487
Project: HBase
Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
Fix For: 0.94.0

Attachments: appendNoSync4.txt, appendNoSync5.txt, appendNoSync6.txt

This allows for better throughput when there are hot rows.I have seen this
change make a single row update improve from 400 increments/sec/server to
4000 increments/sec/server.

[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog

2011-09-30 Thread dhruba borthakur (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118201#comment-13118201
 ] 

dhruba borthakur commented on HBASE-4487:
-

 one cannot guarantee that any individual increment has been committed to the 
 WAL.

Even prior to this patch, individual increments were batched-synced-by-hdfs, so 
there was no guarantee that each increment operation resulted in a sync.

From my understanding, the contract for hbase clients is that an increment 
operation is deemed complete only after it has been sycned to the transaction 
log. This  contract is valid prior to this patch and this patch does not 
change this contract. The server does not return the RPC response back to the 
client until the increment operation has been sycned to the wal. 

Also, the increment operation does not follow the RWCC rules, thereby exposing 
uncommitted increment operations to readers. This patch does not change that 
behaviour either.

is there something I am missing?

 The increment operation can release the rowlock before sync-ing the Hlog
 

 Key: HBASE-4487
 URL: https://issues.apache.org/jira/browse/HBASE-4487
 Project: HBase
  Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: appendNoSync4.txt, appendNoSync5.txt, appendNoSync6.txt


 This allows for better throughput when there are hot rows.I have seen this 
 change make a single row update improve from 400 increments/sec/server to 
 4000 increments/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog

2011-09-30 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118203#comment-13118203
 ] 

Todd Lipcon commented on HBASE-4487:


looking over the patch, I think I agree with Dhruba - dropping the row locks 
before syncing would only have effects on ACID if we were using row locks for 
visibility. But, we use RWCC/MVCC for visibility. So which guarantee are we 
losing?

 The increment operation can release the rowlock before sync-ing the Hlog
 

 Key: HBASE-4487
 URL: https://issues.apache.org/jira/browse/HBASE-4487
 Project: HBase
  Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: appendNoSync4.txt, appendNoSync5.txt, appendNoSync6.txt


 This allows for better throughput when there are hot rows.I have seen this 
 change make a single row update improve from 400 increments/sec/server to 
 4000 increments/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog

2011-09-30 Thread dhruba borthakur (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

dhruba borthakur updated HBASE-4487:


Release Note: The increment operation releases the rowlock before doing the 
sync to the HLog. This improves performance of increments on hot rows. There is 
a fundamental change to the group-commit behaviour: it batches transactions in 
HBase code before pushing it down to the wal.  (was: The increment operation 
releases the rowlock before doing the sync to the HLog. This improves 
performance of increments on hot rows. )

 The increment operation can release the rowlock before sync-ing the Hlog
 

 Key: HBASE-4487
 URL: https://issues.apache.org/jira/browse/HBASE-4487
 Project: HBase
  Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: appendNoSync4.txt, appendNoSync5.txt, appendNoSync6.txt


 This allows for better throughput when there are hot rows.I have seen this 
 change make a single row update improve from 400 increments/sec/server to 
 4000 increments/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers

2011-09-30 Thread Joe Pallas (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118210#comment-13118210
]

Joe Pallas commented on HBASE-4335:
---

@Lars, I think there is evidence that this is not the only way holes in meta
might appear, so I would not mark HBASE-4333 and HBASE-4334 as duplicates. I
think HBASE-4334 is the important one (the server should not assume the client
is correct), and HBASE-4333 is more of an optimization (discover the problem at
the client instead of the server).

Splits can create temporary holes in .META. that confuse clients and
regionservers
--

Key: HBASE-4335
URL: https://issues.apache.org/jira/browse/HBASE-4335
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4
Reporter: Joe Pallas
Assignee: Lars Hofhansl
Priority: Critical
Fix For: 0.92.0

Attachments: 4335-v2.txt, 4335.txt

When a SplitTransaction is performed, three updates are done to .META.:
1. The parent region is marked as splitting (and hence offline)
2. The first daughter region is added (same start key as parent)
3. The second daughter region is added (split key is start key)
(later, the original parent region is deleted, but that's not important to
this discussion)
Steps 2 and 3 are actually done concurrently by
SplitTransaction.DaughterOpener threads. While the master is notified when a
split is complete, the only visibility that clients have is whether the
daughter regions have appeared in .META.
If the second daughter is added to .META. first, then .META. will contain the
(offline) parent region followed by the second daughter region. If the
client looks up a key that is greater than (or equal to) the split, the
client will find the second daughter region and use it. If the key is less
than the split key, the client will find the parent region and see that it is
offline, triggering a retry.
If the first daughter is added to .META. before the second daughter, there is
a window during which .META. has a hole: the first daughter effectively hides
the parent region (same start key), but there is no entry for the second
daughter. A region lookup will find the first daughter for all keys in the
parent's range, but the first daughter does not include keys at or beyond the
split key.
See HBASE-4333 and HBASE-4334 for details on how this causes problems and
suggestions for mitigating this in the client and regionserver.

[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master


[ 
https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118209#comment-13118209
 ] 

jirapos...@reviews.apache.org commented on HBASE-4070:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2029/#review
---



src/main/java/org/apache/hadoop/hbase/HServerLoad.java
https://reviews.apache.org/r/2029/#comment5157

Should return String[]? Trivial to do here given the data is in a Set. Then 
callers can enumerate.



src/main/java/org/apache/hadoop/hbase/HServerLoad.java
https://reviews.apache.org/r/2029/#comment5156

Make a list of unique names for HSL instead of a string?



src/main/java/org/apache/hadoop/hbase/master/HMaster.java
https://reviews.apache.org/r/2029/#comment5155

Should return String[]? Trivial to do here given the data is in a Set.

Then callers can enumerate. Easy enough to display a list of strings in the 
jamon templates. 



src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
https://reviews.apache.org/r/2029/#comment5154

Should return String[]? Then callers can enumerate. Easy enough to display 
a list of strings in the jamon templates.


- Andrew


On 2011-09-30 04:58:20, Eugene Koontz wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2029/
bq.  ---
bq.  
bq.  (Updated 2011-09-30 04:58:20)
bq.  
bq.  
bq.  Review request for hbase and Mingjie Lai.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Proposed fix for HBASE-4070. 
bq.  
bq.  
bq.  This addresses bug HBASE-4070.
bq.  https://issues.apache.org/jira/browse/HBASE-4070
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon 
abeb850 
bq.src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
be6fceb 
bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 01bc1dd 
bq.src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 
bq.src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java 
dbae4fd 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java f80d232 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
96b763b 
bq.
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorReporting.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2029/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Two new tests : testRegionServerCoprocessorReported() and 
testMasterServerCoprocessorsReported() included in a new source file 
src/test/java/o.a.h.h/coprocessor/TestCoprocessorReporting.java.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Eugene
bq.  
bq.



 [Coprocessors] Improve region server metrics to report loaded coprocessors to 
 master
 

 Key: HBASE-4070
 URL: https://issues.apache.org/jira/browse/HBASE-4070
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.3
Reporter: Mingjie Lai
Assignee: Eugene Koontz
 Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, 
 rs-status-web-ui.jpg


 HBASE-3512 is about listing loaded cp classes at shell. To make it more 
 generic, we need a way to report this piece of information from region to 
 master (or just at region server level). So later on, we can display the 
 loaded class names at shell as well as web console. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog

2011-09-30 Thread Andrew Purtell (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118214#comment-13118214
 ] 

Andrew Purtell commented on HBASE-4487:
---

bq. Even prior to this patch, individual increments were 
batched-synced-by-hdfs, so there was no guarantee that each increment operation 
resulted in a sync.

Point taken.

 The increment operation can release the rowlock before sync-ing the Hlog
 

 Key: HBASE-4487
 URL: https://issues.apache.org/jira/browse/HBASE-4487
 Project: HBase
  Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: appendNoSync4.txt, appendNoSync5.txt, appendNoSync6.txt


 This allows for better throughput when there are hot rows.I have seen this 
 change make a single row update improve from 400 increments/sec/server to 
 4000 increments/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master

2011-09-30 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118217#comment-13118217
 ] 

jirapos...@reviews.apache.org commented on HBASE-4070:
--



bq.  On 2011-09-29 22:42:26, Mingjie Lai wrote:
bq.   src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java, 
line 3245
bq.   https://reviews.apache.org/r/2029/diff/2/?file=46422#file46422line3245
bq.  
bq.   is it okay just reuse the existing serverLoad, without rebuilding 
the whole thing? 
bq.   
bq.   this.serverInfo.getLoad()
bq.   
bq.   but it may not be a big issue.

sorry, I don't see a serverInfo member in HRegionServer..?


bq.  On 2011-09-29 22:42:26, Mingjie Lai wrote:
bq.   src/main/java/org/apache/hadoop/hbase/HServerLoad.java, line 190
bq.   https://reviews.apache.org/r/2029/diff/2/?file=46417#file46417line190
bq.  
bq.   an extra indent?

fixed; thanks.


bq.  On 2011-09-29 22:42:26, Mingjie Lai wrote:
bq.   src/main/java/org/apache/hadoop/hbase/HServerLoad.java, line 465
bq.   https://reviews.apache.org/r/2029/diff/2/?file=46417#file46417line465
bq.  
bq.   Just a naming suggestion for walCoprocessors=coprocessors. Although 
right now WALObserver is the only RS level CP, but it'd better not to indicate 
here.

I changed the parameter name to rsCoprocessors and added a javadoc @param 
note; what do you think? Could also just call it coprocessors as you 
suggested. 


bq.  On 2011-09-29 22:42:26, Mingjie Lai wrote:
bq.   src/main/java/org/apache/hadoop/hbase/HServerLoad.java, line 505
bq.   https://reviews.apache.org/r/2029/diff/2/?file=46417#file46417line505
bq.  
bq.   getSimpleName() only returns the class name w/o package info. I 
think we should use the whole class name for the comparator.
bq.  
bq.  Andrew Purtell wrote:
bq.  Full classnames are canonical and avoid accidental conflict, but are 
unwieldy and not so nice to look at. Judgement call, I'd say. I think it 
unlikely that for any given deployment two coprocessors will have the same 
class name but be in a different package.

I would agree with Andrew and use getSimpleName().


bq.  On 2011-09-29 22:42:26, Mingjie Lai wrote:
bq.   src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java, line 261
bq.   https://reviews.apache.org/r/2029/diff/2/?file=46420#file46420line261
bq.  
bq.   The new method is a little odd. how about just getCoprocessors().

Removed HMasterInterface.java changes at Andy's recommendation: now no 
difference with trunk.


bq.  On 2011-09-29 22:42:26, Mingjie Lai wrote:
bq.   src/main/java/org/apache/hadoop/hbase/master/HMaster.java, line 1514
bq.   https://reviews.apache.org/r/2029/diff/2/?file=46421#file46421line1514
bq.  
bq.   still the method name?

changed to getCoprocessors() at Andy's recommendation.


bq.  On 2011-09-29 22:42:26, Mingjie Lai wrote:
bq.   src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java, 
line 3244
bq.   https://reviews.apache.org/r/2029/diff/2/?file=46422#file46422line3244
bq.  
bq.   same as HMasterInterface. can we just generateCoprocessorString() = 
String getCoprocessors()?

changed; thanks.


bq.  On 2011-09-29 22:42:26, Mingjie Lai wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java, line 1652
bq.   https://reviews.apache.org/r/2029/diff/2/?file=46418#file46418line1652
bq.  
bq.   do we really need to provide it thru HTAdmin? if we do, the returned 
value is not correct in case of exception.

removed differences from trunk; you are right: we don't need it here.


bq.  On 2011-09-29 22:42:26, Mingjie Lai wrote:
bq.   
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java, 
line 24
bq.   https://reviews.apache.org/r/2029/diff/2/?file=46424#file46424line24
bq.  
bq.   The file TestCoprocessorEndpoint is particularly for testing cp 
endpoint. It's odd to put the new test cases to this file. Maybe 
TestClassLoading is a better place.

Good idea; moved to a new source file: TestCoprocessorReporting.java.


bq.  On 2011-09-29 22:42:26, Mingjie Lai wrote:
bq.   
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java, 
line 201
bq.   https://reviews.apache.org/r/2029/diff/2/?file=46424#file46424line201
bq.  
bq.   according the previous override comparator in HRL, the cps should be 
in alphabet order. so you don't need to compare this 2 strings.

Thanks; removed extra string and added comment with your point.


- Eugene


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2029/#review2183
---


On 2011-09-30 04:58:20, Eugene Koontz wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.

[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master

2011-09-30 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118218#comment-13118218
 ] 

jirapos...@reviews.apache.org commented on HBASE-4070:
--



bq.  On 2011-09-29 21:52:40, Andrew Purtell wrote:
bq.  
bq.  
bq.  Andrew Purtell wrote:
bq.  Actually, I do have a comment. Do we need a method in HMasterInterface 
for getting the coprocessor set? Can't it be extracted from HSL?
bq.  
bq.  Andrew Purtell wrote:
bq.  I meant via ClusterStatus, therefore via HBaseAdmin.getClusterStatus().

Removed HMasterInterface change since we don't need it.

ClusterStatus constructor has an additional param that is the string 
representation of the Master's loaded coprocessor set.

HBaseAdmin can now use getClusterStatus() to get Master's coprocessor set. 
HBASE-3512's patch will be updated to use this to show master coprocessors in 
the shell.


- Eugene


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2029/#review2179
---


On 2011-09-30 04:58:20, Eugene Koontz wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2029/
bq.  ---
bq.  
bq.  (Updated 2011-09-30 04:58:20)
bq.  
bq.  
bq.  Review request for hbase and Mingjie Lai.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Proposed fix for HBASE-4070. 
bq.  
bq.  
bq.  This addresses bug HBASE-4070.
bq.  https://issues.apache.org/jira/browse/HBASE-4070
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon 
abeb850 
bq.src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
be6fceb 
bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 01bc1dd 
bq.src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 
bq.src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java 
dbae4fd 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java f80d232 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
96b763b 
bq.
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorReporting.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2029/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Two new tests : testRegionServerCoprocessorReported() and 
testMasterServerCoprocessorsReported() included in a new source file 
src/test/java/o.a.h.h/coprocessor/TestCoprocessorReporting.java.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Eugene
bq.  
bq.



 [Coprocessors] Improve region server metrics to report loaded coprocessors to 
 master
 

 Key: HBASE-4070
 URL: https://issues.apache.org/jira/browse/HBASE-4070
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.3
Reporter: Mingjie Lai
Assignee: Eugene Koontz
 Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, 
 rs-status-web-ui.jpg


 HBASE-3512 is about listing loaded cp classes at shell. To make it more 
 generic, we need a way to report this piece of information from region to 
 master (or just at region server level). So later on, we can display the 
 loaded class names at shell as well as web console. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master

2011-09-30 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118220#comment-13118220
 ] 

jirapos...@reviews.apache.org commented on HBASE-4070:
--



bq.  On 2011-09-29 21:52:40, Andrew Purtell wrote:
bq.  
bq.  
bq.  Andrew Purtell wrote:
bq.  Actually, I do have a comment. Do we need a method in HMasterInterface 
for getting the coprocessor set? Can't it be extracted from HSL?
bq.  
bq.  Andrew Purtell wrote:
bq.  I meant via ClusterStatus, therefore via HBaseAdmin.getClusterStatus().
bq.  
bq.  Eugene Koontz wrote:
bq.  Removed HMasterInterface change since we don't need it.
bq.  
bq.  ClusterStatus constructor has an additional param that is the string 
representation of the Master's loaded coprocessor set.
bq.  
bq.  HBaseAdmin can now use getClusterStatus() to get Master's coprocessor 
set. HBASE-3512's patch will be updated to use this to show master coprocessors 
in the shell.

Cool, thanks Eugene


- Andrew


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2029/#review2179
---


On 2011-09-30 04:58:20, Eugene Koontz wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2029/
bq.  ---
bq.  
bq.  (Updated 2011-09-30 04:58:20)
bq.  
bq.  
bq.  Review request for hbase and Mingjie Lai.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Proposed fix for HBASE-4070. 
bq.  
bq.  
bq.  This addresses bug HBASE-4070.
bq.  https://issues.apache.org/jira/browse/HBASE-4070
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon 
abeb850 
bq.src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
be6fceb 
bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 01bc1dd 
bq.src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 
bq.src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java 
dbae4fd 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java f80d232 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
96b763b 
bq.
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorReporting.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2029/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Two new tests : testRegionServerCoprocessorReported() and 
testMasterServerCoprocessorsReported() included in a new source file 
src/test/java/o.a.h.h/coprocessor/TestCoprocessorReporting.java.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Eugene
bq.  
bq.



 [Coprocessors] Improve region server metrics to report loaded coprocessors to 
 master
 

 Key: HBASE-4070
 URL: https://issues.apache.org/jira/browse/HBASE-4070
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.3
Reporter: Mingjie Lai
Assignee: Eugene Koontz
 Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, 
 rs-status-web-ui.jpg


 HBASE-3512 is about listing loaded cp classes at shell. To make it more 
 generic, we need a way to report this piece of information from region to 
 master (or just at region server level). So later on, we can display the 
 loaded class names at shell as well as web console. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master

2011-09-30 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118221#comment-13118221
 ] 

jirapos...@reviews.apache.org commented on HBASE-4070:
--



bq.  On 2011-09-29 22:42:26, Mingjie Lai wrote:
bq.   src/main/java/org/apache/hadoop/hbase/HServerLoad.java, line 465
bq.   https://reviews.apache.org/r/2029/diff/2/?file=46417#file46417line465
bq.  
bq.   Just a naming suggestion for walCoprocessors=coprocessors. Although 
right now WALObserver is the only RS level CP, but it'd better not to indicate 
here.
bq.  
bq.  Eugene Koontz wrote:
bq.  I changed the parameter name to rsCoprocessors and added a javadoc 
@param note; what do you think? Could also just call it coprocessors as you 
suggested.

If the coprocessor list is within a regionserver load report, it's implicitly 
about the regionserver so no rs prefix is needed on the member, no? Seems 
cleaner not to use one.


- Andrew


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2029/#review2183
---


On 2011-09-30 04:58:20, Eugene Koontz wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2029/
bq.  ---
bq.  
bq.  (Updated 2011-09-30 04:58:20)
bq.  
bq.  
bq.  Review request for hbase and Mingjie Lai.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Proposed fix for HBASE-4070. 
bq.  
bq.  
bq.  This addresses bug HBASE-4070.
bq.  https://issues.apache.org/jira/browse/HBASE-4070
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon 
abeb850 
bq.src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
be6fceb 
bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 01bc1dd 
bq.src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 
bq.src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java 
dbae4fd 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java f80d232 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
96b763b 
bq.
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorReporting.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2029/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Two new tests : testRegionServerCoprocessorReported() and 
testMasterServerCoprocessorsReported() included in a new source file 
src/test/java/o.a.h.h/coprocessor/TestCoprocessorReporting.java.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Eugene
bq.  
bq.



 [Coprocessors] Improve region server metrics to report loaded coprocessors to 
 master
 

 Key: HBASE-4070
 URL: https://issues.apache.org/jira/browse/HBASE-4070
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.3
Reporter: Mingjie Lai
Assignee: Eugene Koontz
 Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, 
 rs-status-web-ui.jpg


 HBASE-3512 is about listing loaded cp classes at shell. To make it more 
 generic, we need a way to report this piece of information from region to 
 master (or just at region server level). So later on, we can display the 
 loaded class names at shell as well as web console. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog

2011-09-30 Thread Lars Hofhansl (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118222#comment-13118222
 ] 

Lars Hofhansl commented on HBASE-4487:
--

@ramkrishna and @dhruba. I agree with ramkrishna that the  if (this.closed)  is 
still needed.
True, there already was a race and that was the point of HBASE-4387.
We want to sync without the updateLock held, while at the same make sure we 
don't sync unless we need to (i.e. HLog not closed) and if we fail (which will 
be rare) without the lock we retry with the lock.


 The increment operation can release the rowlock before sync-ing the Hlog
 

 Key: HBASE-4487
 URL: https://issues.apache.org/jira/browse/HBASE-4487
 Project: HBase
  Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: appendNoSync4.txt, appendNoSync5.txt, appendNoSync6.txt


 This allows for better throughput when there are hot rows.I have seen this 
 change make a single row update improve from 400 increments/sec/server to 
 4000 increments/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog

2011-09-30 Thread ramkrishna.s.vasudevan (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118236#comment-13118236
 ] 

ramkrishna.s.vasudevan commented on HBASE-4487:
---

I feel that check will be helpful if at that point of calling syncer() the 
this.closed was true.
But there is a race condition there. 

 The increment operation can release the rowlock before sync-ing the Hlog
 

 Key: HBASE-4487
 URL: https://issues.apache.org/jira/browse/HBASE-4487
 Project: HBase
  Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: appendNoSync4.txt, appendNoSync5.txt, appendNoSync6.txt


 This allows for better throughput when there are hot rows.I have seen this 
 change make a single row update improve from 400 increments/sec/server to 
 4000 increments/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4334) HRegion.get never validates row

2011-09-30 Thread Lars Hofhansl (Assigned) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl reassigned HBASE-4334:


Assignee: Lars Hofhansl

 HRegion.get never validates row
 ---

 Key: HBASE-4334
 URL: https://issues.apache.org/jira/browse/HBASE-4334
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.90.4
Reporter: Joe Pallas
Assignee: Lars Hofhansl

 If a client gets confused (possibly by a hole in .META., see HBASE-4333), it 
 may send a request to the wrong region.  Paths through put, delete, 
 incrementColumnValue, and checkAndMutate all call checkRow either directly or 
 indirectly (through getLock).  But get apparently does not.  This can result 
 in returning an incorrect empty result instead of a WrongRegionException.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers

2011-09-30 Thread Lars Hofhansl (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118237#comment-13118237
]

Lars Hofhansl commented on HBASE-4335:
--

Agreed that HBASE-4334 is important. At least holes would not go silently. I'll
work on that one too.

Splits can create temporary holes in .META. that confuse clients and
regionservers
--

Attachments: 4335-v2.txt, 4335.txt

[jira] [Created] (HBASE-4521) Get the hadoop patch-submission build working for hbase

2011-09-30 Thread stack (Created) (JIRA)

Get the hadoop patch-submission build working for hbase
---

 Key: HBASE-4521
 URL: https://issues.apache.org/jira/browse/HBASE-4521
 Project: HBase
  Issue Type: Task
Reporter: stack


We need the facility over in hadoop where on 'patch submission', jenkins tries 
the patch against current state of trunk.  We need this facility because its a 
productivity killer expecting each dev vet the patch -- let jenkins do it for 
us.  I'm trying to get Giri, the hadoop build fellow, to help us set this up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4357) Region in transition - in closing state

2011-09-30 Thread ramkrishna.s.vasudevan (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118239#comment-13118239
 ] 

ramkrishna.s.vasudevan commented on HBASE-4357:
---

@Ming
3rd point am not sure
But even 2nd point may not be needed right.
Any way the timeout monitor if it sees the state in CLOSING it is again going 
to update to CLOSING.
And  if closeRegion is taking more time due to some bug how do we determine 
after what time we take corrective action?


 Region in transition - in closing state
 ---

 Key: HBASE-4357
 URL: https://issues.apache.org/jira/browse/HBASE-4357
 Project: HBase
  Issue Type: Bug
Reporter: Ming Ma
Assignee: Ming Ma

 Got the following during testing, 
 1. On a given machine, kill RS process id. Then kill HMaster process id.
 2. Start RS first via bin/hbase-daemon.sh --config ./conf start 
 regionserver.. Start HMaster via bin/hbase-daemon.sh --config ./conf start 
 master.
 One region of a table stayed in closing state.
 According to zookeeper,
 794a6ff17a4de0dd0a19b984ba18eea9 
 miweng_500region,H\xB49X\x10bM\xB1,1315338786464.794a6ff17a4de0dd0a19b984ba18eea9.
  state=CLOSING, ts=Wed Sep 07 17:21:44 PDT 2011 (75701s ago), 
 server=sea-esxi-0,6,1315428682281 
 According to .META. table, the region has been assigned to from sea-esxi-0 to 
 sea-esxi-4.
 miweng_500region,H\xB49X\x10bM\xB1,1315338786464.794a6ff17a4de0dd0a19b984ba18eea9.
  sea-esxi-4:60030  H\xB49X\x10bM\xB1 I7K\xC6\xA7\xEF\x9D\x90 0 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4521) Get the hadoop patch-submission build working for hbase

2011-09-30 Thread Jonathan Gray (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118245#comment-13118245
 ] 

Jonathan Gray commented on HBASE-4521:
--

Big +1

 Get the hadoop patch-submission build working for hbase
 ---

 Key: HBASE-4521
 URL: https://issues.apache.org/jira/browse/HBASE-4521
 Project: HBase
  Issue Type: Task
Reporter: stack

 We need the facility over in hadoop where on 'patch submission', jenkins 
 tries the patch against current state of trunk.  We need this facility 
 because its a productivity killer expecting each dev vet the patch -- let 
 jenkins do it for us.  I'm trying to get Giri, the hadoop build fellow, to 
 help us set this up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2794) ROWCOL bloom filter not used if multiple columns within same family are requested in a Get


[ 
https://issues.apache.org/jira/browse/HBASE-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118247#comment-13118247
 ] 

Ted Yu commented on HBASE-2794:
---

TestServerCustomProtocol#testRowRange failed during test suite run but passed 
standalone.

 ROWCOL bloom filter not used if multiple columns within same family are 
 requested in a Get
 --

 Key: HBASE-2794
 URL: https://issues.apache.org/jira/browse/HBASE-2794
 Project: HBase
  Issue Type: Improvement
  Components: performance
Reporter: Kannan Muthukkaruppan
Assignee: Mikhail Bautin
 Fix For: 0.92.0


 Noticed the following snippet in StoreFile.java:Scanner:shouldSeek():
 {code}
 switch(bloomFilterType) {
   case ROW:
 key = row;
 break;
   case ROWCOL:
 if (columns.size() == 1) {
   byte[] col = columns.first();
   key = Bytes.add(row, col);
   break;
 }
 //$FALL-THROUGH$
   default:
 return true;
 }
 {code}
 If columns.size  1, then we currently don't take advantage of the bloom 
 filter.  We should optimize this to check bloom for each of columns and if 
 none of the columns are present in the bloom avoid opening the file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4521) Get the hadoop patch-submission build working for hbase

2011-09-30 Thread Gary Helmling (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118250#comment-13118250
 ] 

Gary Helmling commented on HBASE-4521:
--

+1 This would be really nice to have.

 Get the hadoop patch-submission build working for hbase
 ---

 Key: HBASE-4521
 URL: https://issues.apache.org/jira/browse/HBASE-4521
 Project: HBase
  Issue Type: Task
Reporter: stack

 We need the facility over in hadoop where on 'patch submission', jenkins 
 tries the patch against current state of trunk.  We need this facility 
 because its a productivity killer expecting each dev vet the patch -- let 
 jenkins do it for us.  I'm trying to get Giri, the hadoop build fellow, to 
 help us set this up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4521) Get the hadoop patch-submission build working for hbase

2011-09-30 Thread ramkrishna.s.vasudevan (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118254#comment-13118254
 ] 

Todd Lipcon commented on HBASE-4521:


I've been trying to get Giri to get me access to the new build farm. I used to 
have access, but then the machines got reformatted and no longer do. If I get 
this I can help set this up.

 Get the hadoop patch-submission build working for hbase
 ---

 Key: HBASE-4521
 URL: https://issues.apache.org/jira/browse/HBASE-4521
 Project: HBase
  Issue Type: Task
Reporter: stack

 We need the facility over in hadoop where on 'patch submission', jenkins 
 tries the patch against current state of trunk.  We need this facility 
 because its a productivity killer expecting each dev vet the patch -- let 
 jenkins do it for us.  I'm trying to get Giri, the hadoop build fellow, to 
 help us set this up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml

2011-09-30 Thread Mikhail Bautin (Created) (JIRA)

Make hbase-site-custom.xml override the hbase-site.xml
--

 Key: HBASE-4522
 URL: https://issues.apache.org/jira/browse/HBASE-4522
 Project: HBase
  Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Liyin Tang
Priority: Minor
 Fix For: 0.94.0


The motivation for diff is that we want to override some config change for any 
specific cluster easily by just adding the config entries in the 
hbase-site-custom.xml for that cluster. This change adds the 
hbase-site-custom.xml configuration file into HBaseConfiguration.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog


[ 
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118260#comment-13118260
 ] 

ramkrishna.s.vasudevan commented on HBASE-4487:
---

@Dhruba
I was happy with v6 itself. :).
Just a suggestion it was.
@Ted
:)

 The increment operation can release the rowlock before sync-ing the Hlog
 

 Key: HBASE-4487
 URL: https://issues.apache.org/jira/browse/HBASE-4487
 Project: HBase
  Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: 4487-v7.txt, appendNoSync4.txt, appendNoSync5.txt, 
 appendNoSync6.txt


 This allows for better throughput when there are hot rows.I have seen this 
 change make a single row update improve from 400 increments/sec/server to 
 4000 increments/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-2794) Utilize ROWCOL bloom filter if multiple columns within same family are requested in a Get

2011-09-30 Thread Ted Yu (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-2794:
--

Summary: Utilize ROWCOL bloom filter if multiple columns within same family 
are requested in a Get  (was: ROWCOL bloom filter not used if multiple columns 
within same family are requested in a Get)

 Utilize ROWCOL bloom filter if multiple columns within same family are 
 requested in a Get
 -

 Key: HBASE-2794
 URL: https://issues.apache.org/jira/browse/HBASE-2794
 Project: HBase
  Issue Type: Improvement
  Components: performance
Reporter: Kannan Muthukkaruppan
Assignee: Mikhail Bautin
 Fix For: 0.92.0


 Noticed the following snippet in StoreFile.java:Scanner:shouldSeek():
 {code}
 switch(bloomFilterType) {
   case ROW:
 key = row;
 break;
   case ROWCOL:
 if (columns.size() == 1) {
   byte[] col = columns.first();
   key = Bytes.add(row, col);
   break;
 }
 //$FALL-THROUGH$
   default:
 return true;
 }
 {code}
 If columns.size  1, then we currently don't take advantage of the bloom 
 filter.  We should optimize this to check bloom for each of columns and if 
 none of the columns are present in the bloom avoid opening the file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2794) Utilize ROWCOL bloom filter if multiple columns within same family are requested in a Get

2011-09-30 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118265#comment-13118265
 ] 

Ted Yu commented on HBASE-2794:
---

Integrated to 0.92 branch and TRUNK.

Thanks for the patch Mikhail.

Thanks for the review Jonathan.

 Utilize ROWCOL bloom filter if multiple columns within same family are 
 requested in a Get
 -

 Key: HBASE-2794
 URL: https://issues.apache.org/jira/browse/HBASE-2794
 Project: HBase
  Issue Type: Improvement
  Components: performance
Reporter: Kannan Muthukkaruppan
Assignee: Mikhail Bautin
 Fix For: 0.92.0


 Noticed the following snippet in StoreFile.java:Scanner:shouldSeek():
 {code}
 switch(bloomFilterType) {
   case ROW:
 key = row;
 break;
   case ROWCOL:
 if (columns.size() == 1) {
   byte[] col = columns.first();
   key = Bytes.add(row, col);
   break;
 }
 //$FALL-THROUGH$
   default:
 return true;
 }
 {code}
 If columns.size  1, then we currently don't take advantage of the bloom 
 filter.  We should optimize this to check bloom for each of columns and if 
 none of the columns are present in the bloom avoid opening the file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml

2011-09-30 Thread Jonathan Gray (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118275#comment-13118275
 ] 

Jonathan Gray commented on HBASE-4522:
--

Can't hbase-site import hbase-site-custom?

 Make hbase-site-custom.xml override the hbase-site.xml
 --

 Key: HBASE-4522
 URL: https://issues.apache.org/jira/browse/HBASE-4522
 Project: HBase
  Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Liyin Tang
Priority: Minor
 Fix For: 0.94.0


 The motivation for diff is that we want to override some config change for 
 any specific cluster easily by just adding the config entries in the 
 hbase-site-custom.xml for that cluster. This change adds the 
 hbase-site-custom.xml configuration file into HBaseConfiguration.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4523) dfs.support.append config should be present in the hadoop configs, we should remove them from hbase so the user is not confused when they see the config in 2 places

2011-09-30 Thread Arpit Gupta (Created) (JIRA)

dfs.support.append config should be present in the hadoop configs, we should 
remove them from hbase so the user is not confused when they see the config in 
2 places


 Key: HBASE-4523
 URL: https://issues.apache.org/jira/browse/HBASE-4523
 Project: HBase
  Issue Type: Bug
Reporter: Arpit Gupta




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml

2011-09-30 Thread Liyin Tang (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118284#comment-13118284
 ] 

Liyin Tang commented on HBASE-4522:
---

@Jonathon: It can:) 
That's why I am wondering should I open source this change:)

For us, hbase-site.xml works as hbase-default.xml and hbase-site-custom.xml 
works as hbase-site xml.
That's why we need to make hbase-site-custom.xml overrides to hbase-site.xml.
But in the open source trunk, we don't even have hbase-site-custom.xml at all.

 Make hbase-site-custom.xml override the hbase-site.xml
 --

 Key: HBASE-4522
 URL: https://issues.apache.org/jira/browse/HBASE-4522
 Project: HBase
  Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Liyin Tang
Priority: Minor
 Fix For: 0.94.0


 The motivation for diff is that we want to override some config change for 
 any specific cluster easily by just adding the config entries in the 
 hbase-site-custom.xml for that cluster. This change adds the 
 hbase-site-custom.xml configuration file into HBaseConfiguration.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4523) dfs.support.append config should be present in the hadoop configs, we should remove them from hbase so the user is not confused when they see the config in 2 places

2011-09-30 Thread Eric Yang (Assigned) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Yang reassigned HBASE-4523:


Assignee: Eric Yang

 dfs.support.append config should be present in the hadoop configs, we should 
 remove them from hbase so the user is not confused when they see the config 
 in 2 places
 

 Key: HBASE-4523
 URL: https://issues.apache.org/jira/browse/HBASE-4523
 Project: HBase
  Issue Type: Bug
Reporter: Arpit Gupta
Assignee: Eric Yang



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4334) HRegion.get never validates row

2011-09-30 Thread Lars Hofhansl (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118290#comment-13118290
 ] 

Lars Hofhansl commented on HBASE-4334:
--

I think the best way here is to check the startRow in HRegion.getScanner(...) 
that would cover gets and scans. No scan can reach a region server without the 
startRow set.

 HRegion.get never validates row
 ---

 Key: HBASE-4334
 URL: https://issues.apache.org/jira/browse/HBASE-4334
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.90.4
Reporter: Joe Pallas
Assignee: Lars Hofhansl
 Fix For: 0.92.0


 If a client gets confused (possibly by a hole in .META., see HBASE-4333), it 
 may send a request to the wrong region.  Paths through put, delete, 
 incrementColumnValue, and checkAndMutate all call checkRow either directly or 
 indirectly (through getLock).  But get apparently does not.  This can result 
 in returning an incorrect empty result instead of a WrongRegionException.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4334) HRegion.get never validates row

2011-09-30 Thread Lars Hofhansl (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4334:
-

Fix Version/s: 0.92.0

 HRegion.get never validates row
 ---

 Key: HBASE-4334
 URL: https://issues.apache.org/jira/browse/HBASE-4334
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.90.4
Reporter: Joe Pallas
Assignee: Lars Hofhansl
 Fix For: 0.92.0


 If a client gets confused (possibly by a hole in .META., see HBASE-4333), it 
 may send a request to the wrong region.  Paths through put, delete, 
 incrementColumnValue, and checkAndMutate all call checkRow either directly or 
 indirectly (through getLock).  But get apparently does not.  This can result 
 in returning an incorrect empty result instead of a WrongRegionException.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog

2011-09-30 Thread dhruba borthakur (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118291#comment-13118291
 ] 

dhruba borthakur commented on HBASE-4487:
-

v7 looks good to me, thanks for doing it Ted.

 The increment operation can release the rowlock before sync-ing the Hlog
 

 Key: HBASE-4487
 URL: https://issues.apache.org/jira/browse/HBASE-4487
 Project: HBase
  Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: 4487-v7.txt, appendNoSync4.txt, appendNoSync5.txt, 
 appendNoSync6.txt


 This allows for better throughput when there are hot rows.I have seen this 
 change make a single row update improve from 400 increments/sec/server to 
 4000 increments/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2794) Utilize ROWCOL bloom filter if multiple columns within same family are requested in a Get


[ 
https://issues.apache.org/jira/browse/HBASE-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118294#comment-13118294
 ] 

jirapos...@reviews.apache.org commented on HBASE-2794:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2084/#review2226
---

Ship it!


I'm +0 on commmitting this.  I tried reviewing it but I don't know this code 
well.  The added unit test is nicely intrusive and the asserts look right.  
What about Nicolas's performance concerns.  How are they addressed by this 
patch?  I'm running a build of the patch and if that passes I'm +1 on commit. 


src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
https://reviews.apache.org/r/2084/#comment5175

Interesting method name.  We should use this pattern everywhere we have to 
do this.



src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
https://reviews.apache.org/r/2084/#comment5176

Should we get rid of this javadoc if an override?  (Let us know can do on 
commit)


- Michael


On 2011-09-29 21:05:20, Mikhail Bautin wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2084/
bq.  ---
bq.  
bq.  (Updated 2011-09-29 21:05:20)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Previously we only used row-column Bloom filters for scans that only 
requested one column. We have seen production queries that request up to 200 
columns, and with say ~6 store files per store (region / column family 
combination) this might have resulted in 1200 block read operations in the 
worst case. With this diff we will be avoiding seeks on store files that we 
know don't contain the row/column of interest when using an 
ExplicitColumnTracker. The performance should remain the same for column range 
queries.
bq.  
bq.  
bq.  This addresses bug HBASE-2794.
bq.  https://issues.apache.org/jira/browse/HBASE-2794
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/KeyValue.java 585c4a8 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 
f5173c4 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java a3d778e 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7cbdb98 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java 
9d9895c 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java 
6cdada7 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 4aa72de 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
68cdac5 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
fd9e7ef 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 
08d3ba4 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 
ac2348e 
bq.src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java 
32f88fb 
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueHeap.java 
a5d13f7 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java 
baee696 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2084/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Existing unit tests. A new unit test (TestScanWithBloomError). Load 
testing using HBaseTest.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Mikhail
bq.  
bq.



 Utilize ROWCOL bloom filter if multiple columns within same family are 
 requested in a Get
 -

 Key: HBASE-2794
 URL: https://issues.apache.org/jira/browse/HBASE-2794
 Project: HBase
  Issue Type: Improvement
  Components: performance
Reporter: Kannan Muthukkaruppan
Assignee: Mikhail Bautin
 Fix For: 0.92.0


 Noticed the following snippet in StoreFile.java:Scanner:shouldSeek():
 {code}
 switch(bloomFilterType) {
   case ROW:
 key = row;
 break;
   case ROWCOL:
 if (columns.size() == 1) {
   byte[] col = columns.first();
   key = Bytes.add(row, col);
   break;
 }
 //$FALL-THROUGH$
   default:
 return true;
 }
 {code}
 If columns.size  1,

[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml

2011-09-30 Thread Mikhail Bautin (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118300#comment-13118300
 ] 

Mikhail Bautin commented on HBASE-4522:
---

I discussed this with Kannan, and we thought that this might be good for people 
operating multiple HBase clusters for the same application. In that case 
hbase-site.xml could hold configuration for the entire application and 
hbase-site-custom.xml would have cluster-specific settings, e.g. the ZK quorum.

 Make hbase-site-custom.xml override the hbase-site.xml
 --

 Key: HBASE-4522
 URL: https://issues.apache.org/jira/browse/HBASE-4522
 Project: HBase
  Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Liyin Tang
Priority: Minor
 Fix For: 0.94.0


 The motivation for diff is that we want to override some config change for 
 any specific cluster easily by just adding the config entries in the 
 hbase-site-custom.xml for that cluster. This change adds the 
 hbase-site-custom.xml configuration file into HBaseConfiguration.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog

2011-09-30 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118308#comment-13118308
 ] 

stack commented on HBASE-4487:
--

+1 on v7 (IIUC, Ram and Lars close issue is addressed by the call to sync if 
edits and then the close).  From the back and forth above, my take away is this 
patch makes the grouping fatter but doesn't change current semantics -- we are 
as 'broke' as we were before this patch.  I no longer think this an 
incompatible change.  We should release note the fatter grouping of increments 
on commit. 


 The increment operation can release the rowlock before sync-ing the Hlog
 

 Key: HBASE-4487
 URL: https://issues.apache.org/jira/browse/HBASE-4487
 Project: HBase
  Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: 4487-v7.txt, appendNoSync4.txt, appendNoSync5.txt, 
 appendNoSync6.txt


 This allows for better throughput when there are hot rows.I have seen this 
 change make a single row update improve from 400 increments/sec/server to 
 4000 increments/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog

2011-09-30 Thread Jonathan Gray (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118311#comment-13118311
 ] 

Jonathan Gray commented on HBASE-4487:
--

+1 as well.  And agree with your assessment above, Stack.  Potential fatter 
grouping of increments and significant improvement of per-row throughput.

Looking forward to getting this working for Put/MultiPut!  Nice work, Dhruba.

 The increment operation can release the rowlock before sync-ing the Hlog
 

 Key: HBASE-4487
 URL: https://issues.apache.org/jira/browse/HBASE-4487
 Project: HBase
  Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: 4487-v7.txt, appendNoSync4.txt, appendNoSync5.txt, 
 appendNoSync6.txt


 This allows for better throughput when there are hot rows.I have seen this 
 change make a single row update improve from 400 increments/sec/server to 
 4000 increments/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml

2011-09-30 Thread Mikhail Bautin (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118312#comment-13118312
 ] 

Mikhail Bautin commented on HBASE-4522:
---

Discussed this with Liyin. To clarify Liyin's point above: in our setup, we 
have multiple clusters per application, and hbase-site.xml works as the 
application-specific configuration, while hbase-site-custom.xml is the 
cluster-specific configuration within a particular application. So in our case 
hbase-site.xml does not hard-code any host names. The reason we cannot use 
hbase-default.xml for application-specific configuration is that there is 
already an hbase-default.xml file in the HBase jar, and overriding it would 
make us dependent on the relative ordering of HBase configuration directory and 
the HBase jar in the classpath, which is an unnecessarily brittle approach. 
With the hbase-site-custom.xml file, however, we get additional flexibility in 
configuring our HBase clusters without breaking backwards-compatibility. If the 
hbase-site-custom.xml file is not there, everything will function exactly as 
before.

 Make hbase-site-custom.xml override the hbase-site.xml
 --

 Key: HBASE-4522
 URL: https://issues.apache.org/jira/browse/HBASE-4522
 Project: HBase
  Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Liyin Tang
Priority: Minor
 Fix For: 0.94.0


 The motivation for diff is that we want to override some config change for 
 any specific cluster easily by just adding the config entries in the 
 hbase-site-custom.xml for that cluster. This change adds the 
 hbase-site-custom.xml configuration file into HBaseConfiguration.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class


[ 
https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118323#comment-13118323
 ] 

jirapos...@reviews.apache.org commented on HBASE-4422:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2089/
---

(Updated 2011-09-30 19:36:04.480088)


Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi.


Changes
---

More bug fixes and removed some debug output.

Unit tests are passing for me when I run them in eclipse.  For some reason, 
lots of stuff is frozen when I run it on my remote dev box but having problems 
with and without this patch.

Appreciate reviews and unit test runs!  Thanks


Summary
---

Creates a new CacheConfig class and moves almost everything block cache related 
into this single class.  Adding new configuration params and booleans and such 
should be much better.

All tests are NOT passing yet, still working on it, but wanted to have 
something up today.  Basically code complete but broken :)


This addresses bug HBASE-4422.
https://issues.apache.org/jira/browse/HBASE-4422


Diffs (updated)
-

  
/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java
 1177760 
  /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 
1177760 
  /src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 
1177760 
  
/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
 1177760 
  /src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java 
1177760 
  /src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
1177760 
  /src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java 
1177760 
  
/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java 
1177760 
  /src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java 
1177760 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 1177760 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1177760 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 
1177760 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 
1177760 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 1177760 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 
1177760 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 
1177760 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1177760 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1177760 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1177760 
  /src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 
1177760 
  /src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 
1177760 
  /src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1177760 
  /src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 1177760 
  /src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 
1177760 
  /src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 
1177760 
  /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1177760 
  /src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
1177760 
  /src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1177760 
  /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1177760 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1177760 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1177760 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1177760 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 1177760 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 
1177760 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1177760 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 
1177760 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java PRE-CREATION 
  /src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 1177760 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 
1177760 

Diff: https://reviews.apache.org/r/2089/diff


Testing
---

Still working through some tests that aren't passing.


Thanks,

Jonathan



 Move block cache parameters and references into single CacheConf class
 --

 Key: HBASE-4422
 URL: https://issues.apache.org/jira/browse/HBASE-4422
 Project: HBase

[jira] [Updated] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog


 [ 
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4487:
-

Status: Patch Available  (was: Open)

 The increment operation can release the rowlock before sync-ing the Hlog
 

 Key: HBASE-4487
 URL: https://issues.apache.org/jira/browse/HBASE-4487
 Project: HBase
  Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: 4487-v7.txt, appendNoSync4.txt, appendNoSync5.txt, 
 appendNoSync6.txt


 This allows for better throughput when there are hot rows.I have seen this 
 change make a single row update improve from 400 increments/sec/server to 
 4000 increments/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4334) HRegion.get never validates row


[ 
https://issues.apache.org/jira/browse/HBASE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118325#comment-13118325
 ] 

stack commented on HBASE-4334:
--

Sounds good Lars.  I bet that this bubbles up a bunch of complaints, complaints 
that are legit but that we just rode over clueless in the past.

 HRegion.get never validates row
 ---

 Key: HBASE-4334
 URL: https://issues.apache.org/jira/browse/HBASE-4334
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.90.4
Reporter: Joe Pallas
Assignee: Lars Hofhansl
 Fix For: 0.92.0


 If a client gets confused (possibly by a hole in .META., see HBASE-4333), it 
 may send a request to the wrong region.  Paths through put, delete, 
 incrementColumnValue, and checkAndMutate all call checkRow either directly or 
 indirectly (through getLock).  But get apparently does not.  This can result 
 in returning an incorrect empty result instead of a WrongRegionException.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml


[ 
https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118329#comment-13118329
 ] 

Todd Lipcon commented on HBASE-4522:


Why not just use template-based configuration management here? This seems like 
an HBase feature papering over poor config management, no?

 Make hbase-site-custom.xml override the hbase-site.xml
 --

 Key: HBASE-4522
 URL: https://issues.apache.org/jira/browse/HBASE-4522
 Project: HBase
  Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Liyin Tang
Priority: Minor
 Fix For: 0.94.0


 The motivation for diff is that we want to override some config change for 
 any specific cluster easily by just adding the config entries in the 
 hbase-site-custom.xml for that cluster. This change adds the 
 hbase-site-custom.xml configuration file into HBaseConfiguration.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml

2011-09-30 Thread Mikhail Bautin (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118334#comment-13118334
 ] 

Mikhail Bautin commented on HBASE-4522:
---

@Todd: if I understand correctly, template-based configuration management means 
you generate configuration files using an external site-specific tool. As far 
as I see, there is no standardized tool like that so far in the HBase user 
community (could you please point it out to me if I'm wrong?)  Our approach 
does not require an external template system (it only relies on 
copying/symlinking the right files into the conf directory) and provides a 
clearer separation of between application-specific and site-specific 
configuration. The feature in this JIRA is around two lines of code and does 
not break existing functionality while providing additional flexibility. That 
said, it would be very interesting to hear about alternative approaches to 
Hadoop/HBase configuration management for large multi-cluster deployments -- 
please feel free to share.


 Make hbase-site-custom.xml override the hbase-site.xml
 --

 Key: HBASE-4522
 URL: https://issues.apache.org/jira/browse/HBASE-4522
 Project: HBase
  Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Liyin Tang
Priority: Minor
 Fix For: 0.94.0


 The motivation for diff is that we want to override some config change for 
 any specific cluster easily by just adding the config entries in the 
 hbase-site-custom.xml for that cluster. This change adds the 
 hbase-site-custom.xml configuration file into HBaseConfiguration.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml

2011-09-30 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118341#comment-13118341
]

Todd Lipcon commented on HBASE-4522:

We do configuration management here using puppet. Other deploys Im aware of
use Chef or bcfg2.

It's not that I think the feature is risky, really... it just feels like bloat
and yet another thing to document. Do any other organizations find this feature
useful?

A more general solution that I'd support would be to support something like
hbase/conf.d where any XML files in that directory would be sorted by name,
and loaded in that order. So you could have 01-site.xml, 02-myapplication.xml,
03-my-machine-overrides.xml, etc. This is a more familiar construct for most
sysadmins.

Make hbase-site-custom.xml override the hbase-site.xml
--

Key: HBASE-4522
URL: https://issues.apache.org/jira/browse/HBASE-4522
Project: HBase
Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Liyin Tang
Priority: Minor
Fix For: 0.94.0

The motivation for diff is that we want to override some config change for
any specific cluster easily by just adding the config entries in the
hbase-site-custom.xml for that cluster. This change adds the
hbase-site-custom.xml configuration file into HBaseConfiguration.

[jira] [Updated] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog

2011-09-30 Thread Ted Yu (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4487:
--

Hadoop Flags:   (was: Incompatible change)

 The increment operation can release the rowlock before sync-ing the Hlog
 

 Key: HBASE-4487
 URL: https://issues.apache.org/jira/browse/HBASE-4487
 Project: HBase
  Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: 4487-v7.txt, appendNoSync4.txt, appendNoSync5.txt, 
 appendNoSync6.txt


 This allows for better throughput when there are hot rows.I have seen this 
 change make a single row update improve from 400 increments/sec/server to 
 4000 increments/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml

2011-09-30 Thread Mikhail Bautin (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118380#comment-13118380
 ] 

Mikhail Bautin commented on HBASE-4522:
---

In our case what you call 01-site.xml and 02-myapplication.xml correspond to 
hbase-site.xml, and 03-my-cluster-overrides.xml is what we call 
hbase-site-custom.xml. Right now we only need another level of indirection, so 
implementing the conf.d approach would probably be an overkill for now. We 
definitely need to keep hbase-site.xml, and I am open for suggestions for a 
better name for the next level of customization above hbase-site.xml than 
hbase-site-custom.xml.

We used xi:include to include the per-cluster configuration into 
hbase-site.xml previously, but that created problems when we tried to override 
some properties in hbase-site-custom.xml that were already present in 
hbase-site.xml, since if there are multiple occurrences of the same key in a 
single configuration file, currently there is no guarantee that the earliest or 
the latest one will take precedence, as far as I know. This is why we moved to 
a multi-file approach.

On the feature bloat and documentation side, we are probably looking at two 
lines of code and a few lines of documentation here...


 Make hbase-site-custom.xml override the hbase-site.xml
 --

 Key: HBASE-4522
 URL: https://issues.apache.org/jira/browse/HBASE-4522
 Project: HBase
  Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Liyin Tang
Priority: Minor
 Fix For: 0.94.0


 The motivation for diff is that we want to override some config change for 
 any specific cluster easily by just adding the config entries in the 
 hbase-site-custom.xml for that cluster. This change adds the 
 hbase-site-custom.xml configuration file into HBaseConfiguration.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID


[ 
https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118383#comment-13118383
 ] 

jirapos...@reviews.apache.org commented on HBASE-4507:
--



bq.  On 2011-09-30 01:26:24, Ming Ma wrote:
bq.   Nice stuff, Stack.
bq.   
bq.   Some questions:
bq.   
bq.   1. The approach of isTimestampClash and the nowBytes = 
Bytes.toBytes(now + 1). It seems there is still a rare case, where 
EnvironmentEdgeManager.currentTimeMillis() returns 
expectedTimestampOfLatestVersion - 1. Then later on in put, updateKVTimestamps 
set it to the new now value, which happens to be 
expectedTimestampOfLatestVersion. Perhaps we can use now = 
expectedTimestampOfLatestVersion condition instead of now == 
expectedTimestampOfLatestVersion in isTimestampClash. Set the new time value 
as nowBytes = Bytes.toBytes(expectedTimestampOfLatestVersion + 1).
bq.   2. Do we need to modify coprocessor interfaces preCheckAndPut, 
postCheckAndPut, etc.? Perhaps we don't have any scenario for this yet.
bq.   3. Do we need the same thing for checkAndDelete? Perhaps we don't have 
any scenario for this yet.
bq.  
bq.  Michael Stack wrote:
bq.  1. Doesn't updateKVTimestamps only mess with timestamps if the 
timestamp is set to HConstants.LATEST_TIMESTAMP?  My setting the timestamp here 
before we go down into internalPut means the later updateKVTimestamps will not 
change the timestamps I've set?  Let me check for sure.
bq.  2. Let me look into this.
bq.  3.  I considered it but stopped myself.  Figured I'd let someone ask 
for it before I went and implemented new functionality.
bq.  
bq.  Thanks Ming for review.
bq.

I took another look and now I see what you are saying Ming.  New patch coming.

For 1., for the case where we are all up in the same millisecond, then I'll 
pass the 'now' to the innerPut so that when it does its updateKVTimestamps, its 
operating with the same 'now'.

I fixed 2. adding this new method to RegionObserver and handling the ripple 
through the code base (Man, its harder now adding new methods to a 
regionserver!)

I'm not going to do 3. in scope of this issue.

@Ted New patch will include your suggestion too.

Thanks for review lads.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2118/#review2203
---


On 2011-09-29 23:52:43, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2118/
bq.  ---
bq.  
bq.  (Updated 2011-09-29 23:52:43)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Adds a checkAndPut that takes a timestamp
bq.  
bq.  
bq.  This addresses bug hbase-4507.
bq.  https://issues.apache.org/jira/browse/hbase-4507
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 6ec857c 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
96b763b 
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 
99b34cc 
bq.  
bq.  Diff: https://reviews.apache.org/r/2118/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Michael
bq.  
bq.



 Create checkAndPut variant that exposes timestamp / UUID
 

 Key: HBASE-4507
 URL: https://issues.apache.org/jira/browse/HBASE-4507
 Project: HBase
  Issue Type: Sub-task
Reporter: Ted Yu
Assignee: stack

 Michael checked the checkAndPut which doesn't expose timestamp. A variant of 
 checkAndPut should be created to expose timestamp which is written into a 
 column specified by additional parameters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog

2011-09-30 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118394#comment-13118394
 ] 

Ted Yu commented on HBASE-4487:
---

Integrated patch v7 to TRUNK.

Thanks for the nice feature Dhruba.

Thanks for the reviews from Jonathan, Stack, Lars, Todd, Andrew and Ramkrishna.

 The increment operation can release the rowlock before sync-ing the Hlog
 

 Key: HBASE-4487
 URL: https://issues.apache.org/jira/browse/HBASE-4487
 Project: HBase
  Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: 4487-v7.txt, appendNoSync4.txt, appendNoSync5.txt, 
 appendNoSync6.txt


 This allows for better throughput when there are hot rows.I have seen this 
 change make a single row update improve from 400 increments/sec/server to 
 4000 increments/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml

[
https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118401#comment-13118401
]

jirapos...@reviews.apache.org commented on HBASE-4522:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2135/
---

Review request for Paul Lindner.

Summary
---

The motivation for diff is that we want to override some config change for any
specific cluster easily by just adding the config entries in the
hbase-site-custom.xml for that cluster. This change adds the
hbase-site-custom.xml configuration file into HBaseConfiguration.

This addresses bug HBASE-4522.
https://issues.apache.org/jira/browse/HBASE-4522

Diffs
-

src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java 06fd29c

Diff: https://reviews.apache.org/r/2135/diff

Testing
---

This has been in production on multiple clusters for a few weeks.

Thanks,

Mikhail

Make hbase-site-custom.xml override the hbase-site.xml
--

Key: HBASE-4522
URL: https://issues.apache.org/jira/browse/HBASE-4522
Project: HBase
Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Liyin Tang
Priority: Minor
Fix For: 0.94.0

[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml

2011-09-30 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118405#comment-13118405
]

jirapos...@reviews.apache.org commented on HBASE-4522:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2135/
---

(Updated 2011-09-30 20:30:01.515458)

Review request for hbase, Todd Lipcon and Jonathan Gray.

Changes
---

Changing reviewers -- sorry for spam.

Summary
---

The motivation for diff is that we want to override some config change for any
specific cluster easily by just adding the config entries in the
hbase-site-custom.xml for that cluster. This change adds the
hbase-site-custom.xml configuration file into HBaseConfiguration.

This addresses bug HBASE-4522.
https://issues.apache.org/jira/browse/HBASE-4522

Diffs
-

src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java 06fd29c

Diff: https://reviews.apache.org/r/2135/diff

Testing
---

This has been in production on multiple clusters for a few weeks.

Thanks,

Mikhail

Make hbase-site-custom.xml override the hbase-site.xml
--

Key: HBASE-4522
URL: https://issues.apache.org/jira/browse/HBASE-4522
Project: HBase
Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Liyin Tang
Priority: Minor
Fix For: 0.94.0

[jira] [Commented] (HBASE-4476) Compactions must fail if column tracker gets columns out of order

2011-09-30 Thread Mikhail Bautin (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118429#comment-13118429
 ] 

Mikhail Bautin commented on HBASE-4476:
---

This patch passes all unit tests.

 Compactions must fail if column tracker gets columns out of order
 -

 Key: HBASE-4476
 URL: https://issues.apache.org/jira/browse/HBASE-4476
 Project: HBase
  Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
 Fix For: 0.92.0, 0.94.0


 We found this in ScanWildcardColumnTracker:
 // new col  oldcol
 // if (cmp  0) {
 // WARNING: This means that very likely an edit for some other family
 // was incorrectly stored into the store for this one. Continue, but
 // complain.
 LOG.error(ScanWildcardColumnTracker.checkColumn ran  +
   into a column actually smaller than the previous column:  +
 This went under the radar in our dark launch cluster when a column family 
 name was misspelled first, but then was renamed by renaming directories in 
 the HBase storage directory tree. We ended up with inconsistent data, but 
 compactions still succeeded most of the time, likely discarding part of input 
 data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4476) Compactions must fail if column tracker gets columns out of order

2011-09-30 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118436#comment-13118436
 ] 

jirapos...@reviews.apache.org commented on HBASE-4476:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2050/#review2233
---

Ship it!


- Ted


On 2011-09-29 23:45:46, Mikhail Bautin wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2050/
bq.  ---
bq.  
bq.  (Updated 2011-09-29 23:45:46)
bq.  
bq.  
bq.  Review request for hbase and Kannan Muthukkaruppan.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  We found this in ScanWildcardColumnTracker:
bq.  
bq.  // new col  oldcol
bq.  // if (cmp  0) {
bq.  // WARNING: This means that very likely an edit for some other family
bq.  // was incorrectly stored into the store for this one. Continue, but
bq.  // complain.
bq.  LOG.error(ScanWildcardColumnTracker.checkColumn ran  +
bq.  into a column actually smaller than the previous column:  +
bq.  
bq.  This went under the radar in our dark launch cluster when a column family 
name was misspelled first, but then was renamed by renaming directories in 
the HBase storage directory tree. We ended up with inconsistent data, but 
compactions still succeeded most of the time, likely discarding part of input 
data.
bq.  
bq.  
bq.  This addresses bug HBASE-4476.
bq.  https://issues.apache.org/jira/browse/HBASE-4476
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java
 1176613 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
 1176613 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java
 1176613 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
 1176613 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
 1176613 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java
 1176613 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java
 1176613 
bq.  
bq.  Diff: https://reviews.apache.org/r/2050/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Run unit tests.
bq.  Will also push to a 5-node dev cluster.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Mikhail
bq.  
bq.



 Compactions must fail if column tracker gets columns out of order
 -

 Key: HBASE-4476
 URL: https://issues.apache.org/jira/browse/HBASE-4476
 Project: HBase
  Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
 Fix For: 0.92.0, 0.94.0


 We found this in ScanWildcardColumnTracker:
 // new col  oldcol
 // if (cmp  0) {
 // WARNING: This means that very likely an edit for some other family
 // was incorrectly stored into the store for this one. Continue, but
 // complain.
 LOG.error(ScanWildcardColumnTracker.checkColumn ran  +
   into a column actually smaller than the previous column:  +
 This went under the radar in our dark launch cluster when a column family 
 name was misspelled first, but then was renamed by renaming directories in 
 the HBase storage directory tree. We ended up with inconsistent data, but 
 compactions still succeeded most of the time, likely discarding part of input 
 data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID

2011-09-30 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118440#comment-13118440
]

jirapos...@reviews.apache.org commented on HBASE-4507:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2118/
---

(Updated 2011-09-30 21:03:05.270519)

Review request for hbase.

Changes
---

This version fixes Teds issue and then it does 1. and 2. in Ming's list of 3.

I refactored some moving the timestamp messing down into innerPut. This
cleans up the patch significantly.

Added more javadoc on the method to the effect that this method is for
low-frequency updates on a cell only; i.e. not one edit per millisecond.

What you think Ming?

Summary
---

Adds a checkAndPut that takes a timestamp

This addresses bug hbase-4507.
https://issues.apache.org/jira/browse/hbase-4507

Diffs (updated)
-

src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
96429c7
src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 99b34cc
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b
src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
35377a9
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java 20381f2
src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 6ec857c
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
b05aef2

Diff: https://reviews.apache.org/r/2118/diff

Testing
---

Thanks,

Michael

Create checkAndPut variant that exposes timestamp / UUID

Key: HBASE-4507
URL: https://issues.apache.org/jira/browse/HBASE-4507
Project: HBase
Issue Type: Sub-task
Reporter: Ted Yu
Assignee: stack

Michael checked the checkAndPut which doesn't expose timestamp. A variant of
checkAndPut should be created to expose timestamp which is written into a
column specified by additional parameters.

[jira] [Commented] (HBASE-4496) HFile V2 does not honor setCacheBlocks when scanning.

2011-09-30 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118444#comment-13118444
 ] 

jirapos...@reviews.apache.org commented on HBASE-4496:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2136/#review2232
---

Ship it!


I love it. This is what I should have done with HBASE-4496 if I had had more 
knowledge about the reader code.
I'll do some more manual testing with your patch applied.
This will create extra merging work for HBASE-4422 and HBASE-4344


src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
https://reviews.apache.org/r/2136/#comment5184

This is good.



src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
https://reviews.apache.org/r/2136/#comment5182

I like this. HFileReaderV2 implementing HFileBlock.BasicReader was strange.



src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
https://reviews.apache.org/r/2136/#comment5183

No more casting, awesome.

*Very* minor nit, but why not do 
reader.getDataBlockIndexReader().seekToDataBlock(...) as you do below?


- Lars


On 2011-09-30 20:41:01, Mikhail Bautin wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2136/
bq.  ---
bq.  
bq.  (Updated 2011-09-30 20:41:01)
bq.  
bq.  
bq.  Review request for hbase, Jonathan Gray and Lars Hofhansl.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This fixes a couple of long-existing code issues in HFile v2:
bq.  - Making seekBefore cache the previous block it has to read when the 
scanner happens to be at the first key of a block (this was a performance 
regression introduced in HFile v2).
bq.  - Fixing the accounting of the number of blocks read for the one-level 
index case in HFileBlockIndex.seekToDataBlock if the current block is the same 
as the requested block.
bq.  - Getting rid of HFileBlock.BasicReader, which was used both by FSReaderV2 
and HFileReaderV2, but the former did not cache blocks (a source of confusion).
bq.  - Adding a new interface HFile.CachingBlockReader instead, which is 
implemented by HFile readers and passed to HFileBlockIndex.
bq.  
bq.  
bq.  This addresses bug HBASE-4496.
bq.  https://issues.apache.org/jira/browse/HBASE-4496
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 
4dc1367 
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 
5e98375 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java b429819 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java 
953896e 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 
13d5e70 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 
1cf7767 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java eec566e 
bq.  
bq.  Diff: https://reviews.apache.org/r/2136/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  This is in production in Facebook's hbase-89 branch. 
bq.  
bq.  Still testing this open-source patch -- please don't commit yet.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Mikhail
bq.  
bq.



 HFile V2 does not honor setCacheBlocks when scanning.
 -

 Key: HBASE-4496
 URL: https://issues.apache.org/jira/browse/HBASE-4496
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.92.0, 0.94.0
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.92.0, 0.94.0

 Attachments: 4496.txt


 While testing the LRU cache during the scanning I noticed quite some churn in 
 the cache even when Scan.cacheBlocks is set to false. After debugging this, I 
 found that HFile V2 always caches blocks in the LRU cache regardless of the 
 cacheBlocks setting.
 Here's a trace (from Eclipse) showing the problem:
 HFileReaderV2.readBlock(long, int, boolean, boolean, boolean) line: 279   
 HFileReaderV2.readBlockData(long, long, int, boolean) line: 219   
 HFileBlockIndex$BlockIndexReader.seekToDataBlock(byte[], int, int, 
 HFileBlock) line: 191  
 HFileReaderV2$ScannerV2.seekTo(byte[], int, int, boolean) line: 502   
 HFileReaderV2$ScannerV2.reseekTo(byte[], int, int) line: 539  
 StoreFileScanner.reseekAtOrAfter(HFileScanner, KeyValue) line: 151
 StoreFileScanner.reseek(KeyValue) line: 110   
 KeyValueHeap.reseek(KeyValue) line: 255   
 StoreScanner.reseek(KeyValue) line: 409

[jira] [Updated] (HBASE-4523) dfs.support.append config should be present in the hadoop configs, we should remove them from hbase so the user is not confused when they see the config in 2 places

2011-09-30 Thread Eric Yang (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Yang updated HBASE-4523:
-

Attachment: HBASE-4523.patch

- Updated HBase config template to remove dfs.support.append.

 dfs.support.append config should be present in the hadoop configs, we should 
 remove them from hbase so the user is not confused when they see the config 
 in 2 places
 

 Key: HBASE-4523
 URL: https://issues.apache.org/jira/browse/HBASE-4523
 Project: HBase
  Issue Type: Bug
Reporter: Arpit Gupta
Assignee: Eric Yang
 Attachments: HBASE-4523.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4415) Add configuration script for setup HBase (hbase-setup-conf.sh)

2011-09-30 Thread Eric Yang (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Yang updated HBASE-4415:
-

Affects Version/s: 0.92.0
   0.90.4
Fix Version/s: 0.90.4
   0.92.0

 Add configuration script for setup HBase (hbase-setup-conf.sh)
 --

 Key: HBASE-4415
 URL: https://issues.apache.org/jira/browse/HBASE-4415
 Project: HBase
  Issue Type: New Feature
  Components: scripts
Affects Versions: 0.90.4, 0.92.0
 Environment: Java 6, Linux
Reporter: Eric Yang
Assignee: Eric Yang
 Fix For: 0.90.4, 0.92.0

 Attachments: HBASE-4415-1.patch, HBASE-4415-2.patch, 
 HBASE-4415-3.patch, HBASE-4415-4.patch, HBASE-4415-5.patch, HBASE-4415.patch


 The goal of this jura is to provide a installation script for configuring 
 HBase environment and configuration.  By using the same pattern of 
 *-setup-conf.sh for all Hadoop related projects.  For HBase, the usage of the 
 script looks like this:
 {noformat}
 usage: ./hbase-setup-conf.sh parameters
   Optional parameters:
 --hadoop-conf=/etc/hadoopSet Hadoop configuration directory 
 location
 --hadoop-home=/usr   Set Hadoop directory location
 --hadoop-namenode=localhost  Set Hadoop namenode hostname
 --hadoop-replication=3   Set HDFS replication
 --hbase-home=/usrSet HBase directory location
 --hbase-conf=/etc/hbase  Set HBase configuration 
 directory location
 --hbase-log=/var/log/hbase   Set HBase log directory location
 --hbase-pid=/var/run/hbase   Set HBase pid directory location
 --hbase-user=hbase   Set HBase user
 --java-home=/usr/java/defaultSet JAVA_HOME directory location
 --kerberos-realm=KERBEROS.EXAMPLE.COMSet Kerberos realm
 --kerberos-principal-id=_HOSTSet Kerberos principal ID 
 --keytab-dir=/etc/security/keytabs   Set keytab directory
 --regionservers=localhostSet regionservers hostnames
 --zookeeper-home=/usrSet ZooKeeper directory location
 --zookeeper-quorum=localhost Set ZooKeeper Quorum
 --zookeeper-snapshot=/var/lib/zookeeper  Set ZooKeeper snapshot location
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4496) HFile V2 does not honor setCacheBlocks when scanning.

2011-09-30 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118446#comment-13118446
 ] 

jirapos...@reviews.apache.org commented on HBASE-4496:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2136/#review2234
---



src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
https://reviews.apache.org/r/2136/#comment5186

This is mainly to keep lines 80 characters and for readability.


- Mikhail


On 2011-09-30 20:41:01, Mikhail Bautin wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2136/
bq.  ---
bq.  
bq.  (Updated 2011-09-30 20:41:01)
bq.  
bq.  
bq.  Review request for hbase, Jonathan Gray and Lars Hofhansl.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This fixes a couple of long-existing code issues in HFile v2:
bq.  - Making seekBefore cache the previous block it has to read when the 
scanner happens to be at the first key of a block (this was a performance 
regression introduced in HFile v2).
bq.  - Fixing the accounting of the number of blocks read for the one-level 
index case in HFileBlockIndex.seekToDataBlock if the current block is the same 
as the requested block.
bq.  - Getting rid of HFileBlock.BasicReader, which was used both by FSReaderV2 
and HFileReaderV2, but the former did not cache blocks (a source of confusion).
bq.  - Adding a new interface HFile.CachingBlockReader instead, which is 
implemented by HFile readers and passed to HFileBlockIndex.
bq.  
bq.  
bq.  This addresses bug HBASE-4496.
bq.  https://issues.apache.org/jira/browse/HBASE-4496
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 
4dc1367 
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 
5e98375 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java b429819 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java 
953896e 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 
13d5e70 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 
1cf7767 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java eec566e 
bq.  
bq.  Diff: https://reviews.apache.org/r/2136/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  This is in production in Facebook's hbase-89 branch. 
bq.  
bq.  Still testing this open-source patch -- please don't commit yet.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Mikhail
bq.  
bq.



 HFile V2 does not honor setCacheBlocks when scanning.
 -

 Key: HBASE-4496
 URL: https://issues.apache.org/jira/browse/HBASE-4496
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.92.0, 0.94.0
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.92.0, 0.94.0

 Attachments: 4496.txt


 While testing the LRU cache during the scanning I noticed quite some churn in 
 the cache even when Scan.cacheBlocks is set to false. After debugging this, I 
 found that HFile V2 always caches blocks in the LRU cache regardless of the 
 cacheBlocks setting.
 Here's a trace (from Eclipse) showing the problem:
 HFileReaderV2.readBlock(long, int, boolean, boolean, boolean) line: 279   
 HFileReaderV2.readBlockData(long, long, int, boolean) line: 219   
 HFileBlockIndex$BlockIndexReader.seekToDataBlock(byte[], int, int, 
 HFileBlock) line: 191  
 HFileReaderV2$ScannerV2.seekTo(byte[], int, int, boolean) line: 502   
 HFileReaderV2$ScannerV2.reseekTo(byte[], int, int) line: 539  
 StoreFileScanner.reseekAtOrAfter(HFileScanner, KeyValue) line: 151
 StoreFileScanner.reseek(KeyValue) line: 110   
 KeyValueHeap.reseek(KeyValue) line: 255   
 StoreScanner.reseek(KeyValue) line: 409   
 StoreScanner.next(ListKeyValue, int) line: 304  
 KeyValueHeap.next(ListKeyValue, int) line: 114  
 KeyValueHeap.next(ListKeyValue) line: 143   
 HRegion$RegionScannerImpl.nextRow(byte[]) line: 2774  
 HRegion$RegionScannerImpl.nextInternal(int) line: 2722
 HRegion$RegionScannerImpl.next(ListKeyValue, int) line: 2682
 HRegion$RegionScannerImpl.next(ListKeyValue) line: 2699 
 HRegionServer.next(long, int) line: 2092  
 Every scanner.next causes a reseek, which eventually causes a call to 
 HFileBlockIndex$BlockIndexReader.seekToDataBlock(...) at which point the 
 cacheBlocks information is lost. HFileReaderV2.readBlockData calls

[jira] [Commented] (HBASE-2794) Utilize ROWCOL bloom filter if multiple columns within same family are requested in a Get

2011-09-30 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118449#comment-13118449
 ] 

stack commented on HBASE-2794:
--

These failed after running full suite but seem unrelated:

{code}
Failed tests:   
testOnlineChangeTableSchema(org.apache.hadoop.hbase.client.TestAdmin)
  testForceSplitMultiFamily(org.apache.hadoop.hbase.client.TestAdmin): 
expected:2 but was:1

Tests in error:
  
testEnableDisableAddColumnDeleteColumn(org.apache.hadoop.hbase.client.TestAdmin):
 org.apache.hadoop.hbase.TableNotEnabledException: testMasterAdmin
{code}


 Utilize ROWCOL bloom filter if multiple columns within same family are 
 requested in a Get
 -

 Key: HBASE-2794
 URL: https://issues.apache.org/jira/browse/HBASE-2794
 Project: HBase
  Issue Type: Improvement
  Components: performance
Reporter: Kannan Muthukkaruppan
Assignee: Mikhail Bautin
 Fix For: 0.92.0


 Noticed the following snippet in StoreFile.java:Scanner:shouldSeek():
 {code}
 switch(bloomFilterType) {
   case ROW:
 key = row;
 break;
   case ROWCOL:
 if (columns.size() == 1) {
   byte[] col = columns.first();
   key = Bytes.add(row, col);
   break;
 }
 //$FALL-THROUGH$
   default:
 return true;
 }
 {code}
 If columns.size  1, then we currently don't take advantage of the bloom 
 filter.  We should optimize this to check bloom for each of columns and if 
 none of the columns are present in the bloom avoid opening the file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4476) Compactions must fail if column tracker gets columns out of order


[ 
https://issues.apache.org/jira/browse/HBASE-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118456#comment-13118456
 ] 

jirapos...@reviews.apache.org commented on HBASE-4476:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2050/#review2235
---

Ship it!


- Michael


On 2011-09-29 23:45:46, Mikhail Bautin wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2050/
bq.  ---
bq.  
bq.  (Updated 2011-09-29 23:45:46)
bq.  
bq.  
bq.  Review request for hbase and Kannan Muthukkaruppan.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  We found this in ScanWildcardColumnTracker:
bq.  
bq.  // new col  oldcol
bq.  // if (cmp  0) {
bq.  // WARNING: This means that very likely an edit for some other family
bq.  // was incorrectly stored into the store for this one. Continue, but
bq.  // complain.
bq.  LOG.error(ScanWildcardColumnTracker.checkColumn ran  +
bq.  into a column actually smaller than the previous column:  +
bq.  
bq.  This went under the radar in our dark launch cluster when a column family 
name was misspelled first, but then was renamed by renaming directories in 
the HBase storage directory tree. We ended up with inconsistent data, but 
compactions still succeeded most of the time, likely discarding part of input 
data.
bq.  
bq.  
bq.  This addresses bug HBASE-4476.
bq.  https://issues.apache.org/jira/browse/HBASE-4476
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java
 1176613 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
 1176613 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java
 1176613 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
 1176613 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
 1176613 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java
 1176613 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java
 1176613 
bq.  
bq.  Diff: https://reviews.apache.org/r/2050/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Run unit tests.
bq.  Will also push to a 5-node dev cluster.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Mikhail
bq.  
bq.



 Compactions must fail if column tracker gets columns out of order
 -

 Key: HBASE-4476
 URL: https://issues.apache.org/jira/browse/HBASE-4476
 Project: HBase
  Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
 Fix For: 0.92.0, 0.94.0


 We found this in ScanWildcardColumnTracker:
 // new col  oldcol
 // if (cmp  0) {
 // WARNING: This means that very likely an edit for some other family
 // was incorrectly stored into the store for this one. Continue, but
 // complain.
 LOG.error(ScanWildcardColumnTracker.checkColumn ran  +
   into a column actually smaller than the previous column:  +
 This went under the radar in our dark launch cluster when a column family 
 name was misspelled first, but then was renamed by renaming directories in 
 the HBase storage directory tree. We ended up with inconsistent data, but 
 compactions still succeeded most of the time, likely discarding part of input 
 data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4476) Compactions must fail if column tracker gets columns out of order

2011-09-30 Thread stack (Resolved) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

stack resolved HBASE-4476.
--

Resolution: Fixed
Hadoop Flags: Reviewed

Committed branch and trunk. Thanks for the fixup Mikhail.

Compactions must fail if column tracker gets columns out of order
-

Key: HBASE-4476
URL: https://issues.apache.org/jira/browse/HBASE-4476
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Fix For: 0.92.0, 0.94.0

Attachments: HBASE-4476_v2_2011-09-27_17_38_48.patch

We found this in ScanWildcardColumnTracker:
// new col oldcol
// if (cmp 0) {
// WARNING: This means that very likely an edit for some other family
// was incorrectly stored into the store for this one. Continue, but
// complain.
LOG.error(ScanWildcardColumnTracker.checkColumn ran +
into a column actually smaller than the previous column: +
This went under the radar in our dark launch cluster when a column family
name was misspelled first, but then was renamed by renaming directories in
the HBase storage directory tree. We ended up with inconsistent data, but
compactions still succeeded most of the time, likely discarding part of input
data.

[jira] [Updated] (HBASE-4476) Compactions must fail if column tracker gets columns out of order

2011-09-30 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4476:
-

Attachment: HBASE-4476_v2_2011-09-27_17_38_48.patch

Patch copied from RB.

 Compactions must fail if column tracker gets columns out of order
 -

 Key: HBASE-4476
 URL: https://issues.apache.org/jira/browse/HBASE-4476
 Project: HBase
  Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
 Fix For: 0.92.0, 0.94.0

 Attachments: HBASE-4476_v2_2011-09-27_17_38_48.patch


 We found this in ScanWildcardColumnTracker:
 // new col  oldcol
 // if (cmp  0) {
 // WARNING: This means that very likely an edit for some other family
 // was incorrectly stored into the store for this one. Continue, but
 // complain.
 LOG.error(ScanWildcardColumnTracker.checkColumn ran  +
   into a column actually smaller than the previous column:  +
 This went under the radar in our dark launch cluster when a column family 
 name was misspelled first, but then was renamed by renaming directories in 
 the HBase storage directory tree. We ended up with inconsistent data, but 
 compactions still succeeded most of the time, likely discarding part of input 
 data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4377) [hbck] Offline rebuild .META. from fs data only.


[ 
https://issues.apache.org/jira/browse/HBASE-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118468#comment-13118468
 ] 

jirapos...@reviews.apache.org commented on HBASE-4377:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2126/#review2231
---


How long did it take to scan the cluster with 2700 inconsistencies ?
I see certain places in the code where more parallelism can be achieved if 
practical use of this feature takes long time.


src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
https://reviews.apache.org/r/2126/#comment5180

Better replace root with -ROOT-



src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
https://reviews.apache.org/r/2126/#comment5179

b is not needed here, same with question mark.



src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
https://reviews.apache.org/r/2126/#comment5181

Please remove b and question mark.



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/2126/#comment5185

It would be nice to log the path for the underlying region.
Otherwise what purpose does this catch/rethrow serve ?



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/2126/#comment5188

The .META. region is open upon return.
I think we should document this.



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/2126/#comment5187

Looking at the usage below, maybe createNewRootAndMeta would be a better 
name.



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/2126/#comment5189

This log doesn't match the check above.
If we only produce Put for the first HbckInfo, we'd better declare that in 
the log.



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/2126/#comment5190

This would produce exception if his.size() == 0.



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/2126/#comment5192

Better use boolean for return value to indicate success/failure.



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/2126/#comment5191

Do you plan to do this in the next patch or in another JIRA ?
I haven't looked at the other JIRAs you mentioned, pardon me.



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/2126/#comment5193

Is there something we can do in case we get IOE from this call ?


- Ted


On 2011-09-30 00:02:16, jmhsieh wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2126/
bq.  ---
bq.  
bq.  (Updated 2011-09-30 00:02:16)
bq.  
bq.  
bq.  Review request for hbase, Michael Stack and Andrew Purtell.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  commit fbf82c17be6b3ecca5a981f5270cf93aac26e479
bq.  Author: Jonathan Hsieh j...@cloudera.com
bq.  Date:   Wed Sep 28 10:18:11 2011 -0700
bq.  
bq.  HBASE-4377 [hbck] Offline rebuild .META. from fs data only
bq.  
bq.  
bq.  This patch rebuilds a new .META. table by reading all the .regioninfo 
files in the hbase main directory.  It depends on the yet to be committed 
HBASE-4515 (either my verison or Gary's version), HBASE-4509, and HBASE-4506.  
bq.  
bq.  Some follow on work includes backporting to 0.90, auto-patching true 
holes, and adding documentation.
bq.  
bq.  
bq.  This addresses bug HBASE-4377.
bq.  https://issues.apache.org/jira/browse/HBASE-4377
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 
b9c850d 
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 8465724 
bq.src/main/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRepair.java 
PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java fae0881 
bq.src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java 
PRE-CREATION 
bq.
src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuild.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2126/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  An earlier version of this code (backported to 0.90) was used to diagnose 
and repair a cluster that had 2700 inconsistencies due to failed splits (the 
cluster was underprovisioned memory-wise, and on restart, the some regions 
would start splitting and then die due to oome's).  This was not actually used 
on a live cluster -- it was used to reconstruct a .META. from .regioninfo's 
laid out in

[jira] [Commented] (HBASE-2794) Utilize ROWCOL bloom filter if multiple columns within same family are requested in a Get

2011-09-30 Thread Mikhail Bautin (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118471#comment-13118471
 ] 

Mikhail Bautin commented on HBASE-2794:
---

@Michael: I am observing a different set of spuriously failing tests, also 
seemingly unrelated.

2011-09-29_20_41_15 | tests: 1015, fail: 0, err: 0, skip: 21, time: 6027.3
2011-09-29_23_09_51 | tests: 1012, fail: 0, err: 0, skip: 21, time: 5328.0
2011-09-30_01_44_42 | tests: 1015, fail: 0, err: 0, skip: 21, time: 6338.4
2011-09-30_04_28_29 | tests: 1015, fail: 0, err: 0, skip: 21, time: 6079.2
2011-09-30_07_00_24 | tests: 1015, fail: 1, err: 0, skip: 21, time: 6656.2, 
failed: Admin
2011-09-30_09_41_53 | tests: 1015, fail: 0, err: 0, skip: 21, time: 5900.8
2011-09-30_12_10_25 | tests: 1004, fail: 1, err: 0, skip: 21, time: 5397.7, 
failed: DistributedLogSplitting

(Patch applied on top of http://svn.apache.org/repos/asf/hbase/trunk@1176613)


 Utilize ROWCOL bloom filter if multiple columns within same family are 
 requested in a Get
 -

 Key: HBASE-2794
 URL: https://issues.apache.org/jira/browse/HBASE-2794
 Project: HBase
  Issue Type: Improvement
  Components: performance
Reporter: Kannan Muthukkaruppan
Assignee: Mikhail Bautin
 Fix For: 0.92.0


 Noticed the following snippet in StoreFile.java:Scanner:shouldSeek():
 {code}
 switch(bloomFilterType) {
   case ROW:
 key = row;
 break;
   case ROWCOL:
 if (columns.size() == 1) {
   byte[] col = columns.first();
   key = Bytes.add(row, col);
   break;
 }
 //$FALL-THROUGH$
   default:
 return true;
 }
 {code}
 If columns.size  1, then we currently don't take advantage of the bloom 
 filter.  We should optimize this to check bloom for each of columns and if 
 none of the columns are present in the bloom avoid opening the file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4503) Purge deprecated HBaseClusterTestCase

2011-09-30 Thread stack (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-4503.
--

   Resolution: Fixed
Fix Version/s: 0.92.0
 Hadoop Flags: Reviewed

Committed to 0.92 and to trunk.  Thanks for reviews Jesse, Ram, and Ted.

 Purge deprecated HBaseClusterTestCase
 -

 Key: HBASE-4503
 URL: https://issues.apache.org/jira/browse/HBASE-4503
 Project: HBase
  Issue Type: Improvement
Reporter: stack
Assignee: stack
 Fix For: 0.92.0

 Attachments: 4503-v2.txt, 4503.txt


 It could gain us a few minutes on overall test run in the cases where we 
 don't spin up a cluster for each test.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4377) [hbck] Offline rebuild .META. from fs data only.

2011-09-30 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118480#comment-13118480
 ] 

jirapos...@reviews.apache.org commented on HBASE-4377:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2126/#review2236
---



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/2126/#comment5194

My comment above on the first rename call was inaccurate.
IOE out of the second call would be fatal.


- Ted


On 2011-09-30 00:02:16, jmhsieh wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2126/
bq.  ---
bq.  
bq.  (Updated 2011-09-30 00:02:16)
bq.  
bq.  
bq.  Review request for hbase, Michael Stack and Andrew Purtell.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  commit fbf82c17be6b3ecca5a981f5270cf93aac26e479
bq.  Author: Jonathan Hsieh j...@cloudera.com
bq.  Date:   Wed Sep 28 10:18:11 2011 -0700
bq.  
bq.  HBASE-4377 [hbck] Offline rebuild .META. from fs data only
bq.  
bq.  
bq.  This patch rebuilds a new .META. table by reading all the .regioninfo 
files in the hbase main directory.  It depends on the yet to be committed 
HBASE-4515 (either my verison or Gary's version), HBASE-4509, and HBASE-4506.  
bq.  
bq.  Some follow on work includes backporting to 0.90, auto-patching true 
holes, and adding documentation.
bq.  
bq.  
bq.  This addresses bug HBASE-4377.
bq.  https://issues.apache.org/jira/browse/HBASE-4377
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 
b9c850d 
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 8465724 
bq.src/main/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRepair.java 
PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java fae0881 
bq.src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java 
PRE-CREATION 
bq.
src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuild.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2126/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  An earlier version of this code (backported to 0.90) was used to diagnose 
and repair a cluster that had 2700 inconsistencies due to failed splits (the 
cluster was underprovisioned memory-wise, and on restart, the some regions 
would start splitting and then die due to oome's).  This was not actually used 
on a live cluster -- it was used to reconstruct a .META. from .regioninfo's 
laid out in hbase's directory structure.
bq.  
bq.  Note also that this is not an automatic fix -- whenever any problems are 
found, this bails out but dumps info on holes, suggests some fixes, and 
displays sets of overlapping regions.  It is up to the user to merge regions, 
to create .regioninfo files to plug hole, and to do any potential data loosing 
operations.
bq.  
bq.  The tests demonstrate current expected behavior -- rebuild meta if things 
line up, and fail without making modifications if holes or overlaps exist.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  jmhsieh
bq.  
bq.



 [hbck] Offline rebuild .META. from fs data only.
 

 Key: HBASE-4377
 URL: https://issues.apache.org/jira/browse/HBASE-4377
 Project: HBase
  Issue Type: New Feature
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: 
 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.patch


 In a worst case situation, it may be helpful to have an offline .META. 
 rebuilder that just looks at the file system's .regioninfos and rebuilds meta 
 from scratch.  Users could move bad regions out until there is a clean 
 rebuild.  
 It would likely fill in region split holes.  Follow on work could given 
 options to merge or select regions that overlap, or do online rebuilds.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID


[ 
https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118481#comment-13118481
 ] 

Todd Lipcon commented on HBASE-4507:


Do we really need to modify checkAndPut here? I think it's safer to have 
explicit sequence IDs for the META rows and use the existing checkAndPut. 
Unless there's some reason that won't work, I don't think it's necessary to 
change a core function for this?

 Create checkAndPut variant that exposes timestamp / UUID
 

 Key: HBASE-4507
 URL: https://issues.apache.org/jira/browse/HBASE-4507
 Project: HBase
  Issue Type: Sub-task
Reporter: Ted Yu
Assignee: stack

 Michael checked the checkAndPut which doesn't expose timestamp. A variant of 
 checkAndPut should be created to expose timestamp which is written into a 
 column specified by additional parameters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4209) The HBase hbase-daemon.sh SIGKILLs master when stopping it


 [ 
https://issues.apache.org/jira/browse/HBASE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4209:
-

Attachment: 4209-v3.txt

Patch with the Todd fix and I swapped an unorthodox public AFTER a static 
qualifier.  I tried this patch and it works for me.  Committing this.

 The HBase hbase-daemon.sh SIGKILLs master when stopping it
 --

 Key: HBASE-4209
 URL: https://issues.apache.org/jira/browse/HBASE-4209
 Project: HBase
  Issue Type: Bug
  Components: master
Reporter: Roman Shaposhnik
Assignee: Roman Shaposhnik
 Attachments: 4209-v3.txt, HBASE-4209.final.patch.txt, 
 HBASE-4209.patch.txt


 There's a bit of code in hbase-daemon.sh that makes HBase master being 
 SIGKILLed when stopping it rather than trying SIGTERM (like it does for other 
 daemons). When HBase is executed in a standalone mode (and the only daemon 
 you need to run is master) that causes newly created tables to go missing as 
 unflushed data is thrown out. If there was not a good reason to kill master 
 with SIGKILL perhaps we can take that special case out and rely on SIGTERM.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4209) The HBase hbase-daemon.sh SIGKILLs master when stopping it


 [ 
https://issues.apache.org/jira/browse/HBASE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4209:
-

   Resolution: Fixed
Fix Version/s: 0.92.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed to branch 0.92 and trunk.  Thank you for the patch Roman.

 The HBase hbase-daemon.sh SIGKILLs master when stopping it
 --

 Key: HBASE-4209
 URL: https://issues.apache.org/jira/browse/HBASE-4209
 Project: HBase
  Issue Type: Bug
  Components: master
Reporter: Roman Shaposhnik
Assignee: Roman Shaposhnik
 Fix For: 0.92.0

 Attachments: 4209-v3.txt, HBASE-4209.final.patch.txt, 
 HBASE-4209.patch.txt


 There's a bit of code in hbase-daemon.sh that makes HBase master being 
 SIGKILLed when stopping it rather than trying SIGTERM (like it does for other 
 daemons). When HBase is executed in a standalone mode (and the only daemon 
 you need to run is master) that causes newly created tables to go missing as 
 unflushed data is thrown out. If there was not a good reason to kill master 
 with SIGKILL perhaps we can take that special case out and rely on SIGTERM.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID


[ 
https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118487#comment-13118487
 ] 

Todd Lipcon commented on HBASE-4507:


If we're going to add more functionality to checkAndPut I'd really rather do it 
with a clean API, like:

{code}
boolean atomicMutation(RowPredicates predicates, Action mutation);
{code}
used something like:
{code}
RowPredicates pred = new RowPredicates.Builder()
  .matchTimestamp(ComparisonOp.EQUALS, 12345)
  .matchCellValue(colFam, qual, ComparisonOp.EQUALS, val)
  .build();
Action mutation = new Put(...);
table.atomicMutation(pred, mutation);
{code}

so we don't have to add a new checkAndPut API for every combinatoric set of 
options and mutations

 Create checkAndPut variant that exposes timestamp / UUID
 

 Key: HBASE-4507
 URL: https://issues.apache.org/jira/browse/HBASE-4507
 Project: HBase
  Issue Type: Sub-task
Reporter: Ted Yu
Assignee: stack

 Michael checked the checkAndPut which doesn't expose timestamp. A variant of 
 checkAndPut should be created to expose timestamp which is written into a 
 column specified by additional parameters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID


[ 
https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118492#comment-13118492
 ] 

Todd Lipcon commented on HBASE-4507:


(actually, reusing Filter for this may make sense)

 Create checkAndPut variant that exposes timestamp / UUID
 

 Key: HBASE-4507
 URL: https://issues.apache.org/jira/browse/HBASE-4507
 Project: HBase
  Issue Type: Sub-task
Reporter: Ted Yu
Assignee: stack

 Michael checked the checkAndPut which doesn't expose timestamp. A variant of 
 checkAndPut should be created to expose timestamp which is written into a 
 column specified by additional parameters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4436) Remove methods deprecated in 0.90 from TRUNK and 0.92

2011-09-30 Thread Jonathan Hsieh (Assigned) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh reassigned HBASE-4436:
-

Assignee: Jonathan Hsieh

 Remove methods deprecated in 0.90 from TRUNK and 0.92
 -

 Key: HBASE-4436
 URL: https://issues.apache.org/jira/browse/HBASE-4436
 Project: HBase
  Issue Type: Task
Reporter: stack
Assignee: Jonathan Hsieh
Priority: Critical
  Labels: noob
 Fix For: 0.92.0


 Remove methods deprecated in 0.90 from codebase.  i took a quick look.  The 
 messy bit is thrift referring to old stuff; will take a little work to do the 
 convertion over to the new methods.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4385) Use CacheBuilder in place of MapMaker


[ 
https://issues.apache.org/jira/browse/HBASE-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118494#comment-13118494
 ] 

Ted Yu commented on HBASE-4385:
---

Guava 10.0 doesn't support Cache.put()
We need to wait for 10.1 or 11.0

 Use CacheBuilder in place of MapMaker
 -

 Key: HBASE-4385
 URL: https://issues.apache.org/jira/browse/HBASE-4385
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
 Attachments: 4385.txt


 Guava release 10 introduced CacheBuilder.
 We should use it in place of MapMaker which is used for caching.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4436) Remove methods deprecated in 0.90 from TRUNK and 0.92

2011-09-30 Thread Jonathan Hsieh (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118495#comment-13118495
 ] 

Jonathan Hsieh commented on HBASE-4436:
---

I'll take this.  I'm going to break it down to a few patches.  The first will 
be the completely trivial changes, to be followed by a series of more 
complicated patches.  

Is removing classes like HServerAddress in scope? (It is pretty pervasive)

 Remove methods deprecated in 0.90 from TRUNK and 0.92
 -

 Key: HBASE-4436
 URL: https://issues.apache.org/jira/browse/HBASE-4436
 Project: HBase
  Issue Type: Task
Reporter: stack
Assignee: Jonathan Hsieh
Priority: Critical
  Labels: noob
 Fix For: 0.92.0


 Remove methods deprecated in 0.90 from codebase.  i took a quick look.  The 
 messy bit is thrift referring to old stuff; will take a little work to do the 
 convertion over to the new methods.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID

[
https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118498#comment-13118498
]

stack commented on HBASE-4507:
--

bq. Do we really need to modify checkAndPut here?

checkAndPut as is has a hole in that an update that writes the same value will
go unnoticed.

bq. I think it's safer to have explicit sequence IDs for the META rows and use
the existing checkAndPut. Unless there's some reason that won't work, I don't
think it's necessary to change a core function for this?

So, each time we set a region location (hostname + port + startcode), we'd also
update a new column, say info:editid, which would have a uuid value. We'd then
do checkAndPut on the info:editid column?

We'd still need to read whatever this value was at start of region open.

I could work up a patch for this over in hbase-4497 if its preferred.

Regards atomicMutation, that would be a nice improvement. Would deprecate
current APIs in favor of it going forward. Lets open a new issue to do that.

Create checkAndPut variant that exposes timestamp / UUID

Key: HBASE-4507
URL: https://issues.apache.org/jira/browse/HBASE-4507
Project: HBase
Issue Type: Sub-task
Reporter: Ted Yu
Assignee: stack

Michael checked the checkAndPut which doesn't expose timestamp. A variant of
checkAndPut should be created to expose timestamp which is written into a
column specified by additional parameters.

[jira] [Commented] (HBASE-4436) Remove methods deprecated in 0.90 from TRUNK and 0.92

2011-09-30 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118499#comment-13118499
 ] 

stack commented on HBASE-4436:
--

bq. Is removing classes like HServerAddress in scope? (It is pretty pervasive)

Be careful.  HSA is deprecated for 0.92.  You'll need to look at what was 
deprecated in 0.90 and only remove those for 0.92.

Thanks Jon.

 Remove methods deprecated in 0.90 from TRUNK and 0.92
 -

 Key: HBASE-4436
 URL: https://issues.apache.org/jira/browse/HBASE-4436
 Project: HBase
  Issue Type: Task
Reporter: stack
Assignee: Jonathan Hsieh
Priority: Critical
  Labels: noob
 Fix For: 0.92.0


 Remove methods deprecated in 0.90 from codebase.  i took a quick look.  The 
 messy bit is thrift referring to old stuff; will take a little work to do the 
 convertion over to the new methods.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID


[ 
https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118510#comment-13118510
 ] 

jirapos...@reviews.apache.org commented on HBASE-4507:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2118/#review2238
---

Ship it!


It looks good. BTW, it looks like we don't have unit tests for 
RegionObserver.preCheckAndPut and RegionObserver.postCheckAndPut. Maybe need a 
separate jira for that.

- Ming


On 2011-09-30 21:03:05, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2118/
bq.  ---
bq.  
bq.  (Updated 2011-09-30 21:03:05)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Adds a checkAndPut that takes a timestamp
bq.  
bq.  
bq.  This addresses bug hbase-4507.
bq.  https://issues.apache.org/jira/browse/hbase-4507
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
 96429c7 
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 
99b34cc 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
96b763b 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java 
35377a9 
bq.src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java 
20381f2 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 6ec857c 
bq.
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java 
b05aef2 
bq.  
bq.  Diff: https://reviews.apache.org/r/2118/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Michael
bq.  
bq.



 Create checkAndPut variant that exposes timestamp / UUID
 

 Key: HBASE-4507
 URL: https://issues.apache.org/jira/browse/HBASE-4507
 Project: HBase
  Issue Type: Sub-task
Reporter: Ted Yu
Assignee: stack

 Michael checked the checkAndPut which doesn't expose timestamp. A variant of 
 checkAndPut should be created to expose timestamp which is written into a 
 column specified by additional parameters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4418) Show all the hbase configuration in the web ui