[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors

2012-07-27 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423694#comment-13423694
 ] 

Hadoop QA commented on HBASE-6427:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12538127/6427-v3.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 27 new or modified tests.

+1 hadoop2.0.  The patch compiles against the hadoop 2.0 profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

-1 javac.  The applied patch generated 5 javac compiler warnings (more than 
the trunk's current 4 warnings).

-1 findbugs.  The patch appears to introduce 14 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.replication.TestReplication
  org.apache.hadoop.hbase.master.TestAssignmentManager

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2445//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2445//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2445//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2445//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2445//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2445//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2445//console

This message is automatically generated.

 Pluggable compaction policies via coprocessors
 --

 Key: HBASE-6427
 URL: https://issues.apache.org/jira/browse/HBASE-6427
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt, 6427-v3.txt


 When implementing higher level stores on top of HBase it is necessary to 
 allow dynamic control over how long KVs must be kept around.
 Semi-static config options for ColumnFamilies (# of version or TTL) is not 
 sufficient.
 This can be done with a few additional coprocessor hooks, or by makeing 
 Store.ScanInfo pluggable.
 Was:
 The simplest way to achieve this is to have a pluggable class to determine 
 the smallestReadpoint for Region. That way outside code can control what KVs 
 to retain.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6461) Killing the HRegionServer and DataNode hosting ROOT can result in a malformed root table.

2012-07-27 Thread Lars Hofhansl (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423695#comment-13423695
 ] 

Lars Hofhansl commented on HBASE-6461:
--

I thought HBase neither uses nor needs append (the reason for initial using 
the append branch was that it also had the sync patches).
Where do we ever rely on a reader and a writer concurrently 
reader-from/writing-to the same file (Hlog in this case).
Is this only a special case for the WAL when we start to replay the edits, when 
the file is not closed, yet, because the writer died?


 Killing the HRegionServer and DataNode hosting ROOT can result in a malformed 
 root table.
 -

 Key: HBASE-6461
 URL: https://issues.apache.org/jira/browse/HBASE-6461
 Project: HBase
  Issue Type: Bug
 Environment: hadoop-0.20.2-cdh3u3
 HBase 0.94.1 RC1
Reporter: Elliott Clark
Priority: Critical
 Fix For: 0.94.2


 Spun up a new dfs on hadoop-0.20.2-cdh3u3
 Started hbase
 started running loadtest tool.
 killed rs and dn holding root with killall -9 java on server sv4r27s44 at 
 about 2012-07-25 22:40:00
 After things stabilize Root is in a bad state. Ran hbck and got:
 Exception in thread main 
 org.apache.hadoop.hbase.client.NoServerForRegionException: No server address 
 listed in -ROOT- for region .META.,,1.1028785192 containing row 
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1016)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
 at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232)
 at org.apache.hadoop.hbase.client.HTable.init(HTable.java:172)
 at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:241)
 at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3236)
 hbase(main):001:0 scan '-ROOT-'
 ROW   COLUMN+CELL 
   
 
 12/07/25 22:43:18 INFO security.UserGroupInformation: JAAS Configuration 
 already set up for Hadoop, not re-installing.
  .META.,,1column=info:regioninfo, 
 timestamp=1343255838525, value={NAME = '.META.,,1', STARTKEY = '', ENDKEY 
 = '', ENCODED = 1028785192,}
  .META.,,1column=info:v, 
 timestamp=1343255838525, value=\x00\x00   
  
 1 row(s) in 0.5930 seconds
 Here's the master log: https://gist.github.com/3179194
 I tried the same thing with 0.92.1 and I was able to get into a similar 
 situation, so I don't think this is anything new. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6372) Add scanner batching to Export job

2012-07-27 Thread Shengsheng Huang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423698#comment-13423698
 ] 

Shengsheng Huang commented on HBASE-6372:
-

{quote}
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort
 
{quote}
I don't think this patch has something to do with this unit test.  This unit 
test had passed at my local deployment. 

 Add scanner batching to Export job
 --

 Key: HBASE-6372
 URL: https://issues.apache.org/jira/browse/HBASE-6372
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Affects Versions: 0.96.0, 0.94.2
Reporter: Lars George
Priority: Minor
  Labels: newbie
 Attachments: HBASE-6372.patch


 When a single row is too large for the RS heap then an OOME can take out the 
 entire RS. Setting scanner batching in custom scans helps avoiding this 
 scenario, but for the supplied Export job this is not set.
 Similar to HBASE-3421 we can set the batching to a low number - or if needed 
 make it a command line option.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6461) Killing the HRegionServer and DataNode hosting ROOT can result in a malformed root table.

2012-07-27 Thread Elliott Clark (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423700#comment-13423700
 ] 

Elliott Clark commented on HBASE-6461:
--

@nkeywal
I'm not seeing that log exception.  However everything else is exactly the 
same.  This only shows when I kill the DN and RS at the same time.  And does 
seem to manifest itself more when the log splitting starts very soon after the 
java processes are killed.

 Killing the HRegionServer and DataNode hosting ROOT can result in a malformed 
 root table.
 -

 Key: HBASE-6461
 URL: https://issues.apache.org/jira/browse/HBASE-6461
 Project: HBase
  Issue Type: Bug
 Environment: hadoop-0.20.2-cdh3u3
 HBase 0.94.1 RC1
Reporter: Elliott Clark
Priority: Critical
 Fix For: 0.94.2


 Spun up a new dfs on hadoop-0.20.2-cdh3u3
 Started hbase
 started running loadtest tool.
 killed rs and dn holding root with killall -9 java on server sv4r27s44 at 
 about 2012-07-25 22:40:00
 After things stabilize Root is in a bad state. Ran hbck and got:
 Exception in thread main 
 org.apache.hadoop.hbase.client.NoServerForRegionException: No server address 
 listed in -ROOT- for region .META.,,1.1028785192 containing row 
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1016)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
 at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232)
 at org.apache.hadoop.hbase.client.HTable.init(HTable.java:172)
 at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:241)
 at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3236)
 hbase(main):001:0 scan '-ROOT-'
 ROW   COLUMN+CELL 
   
 
 12/07/25 22:43:18 INFO security.UserGroupInformation: JAAS Configuration 
 already set up for Hadoop, not re-installing.
  .META.,,1column=info:regioninfo, 
 timestamp=1343255838525, value={NAME = '.META.,,1', STARTKEY = '', ENDKEY 
 = '', ENCODED = 1028785192,}
  .META.,,1column=info:v, 
 timestamp=1343255838525, value=\x00\x00   
  
 1 row(s) in 0.5930 seconds
 Here's the master log: https://gist.github.com/3179194
 I tried the same thing with 0.92.1 and I was able to get into a similar 
 situation, so I don't think this is anything new. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6456) Export Utility throws ClassNotFoundException

2012-07-27 Thread Shengsheng Huang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423705#comment-13423705
 ] 

Shengsheng Huang commented on HBASE-6456:
-

{quote}
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestFromClientSide
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort
org.apache.hadoop.hbase.catalog.TestMetaReaderEditor
org.apache.hadoop.hbase.master.TestMasterNoCluster
org.apache.hadoop.hbase.security.access.TestZKPermissionsWatcher
{quote}

I don't think this patch has something to do with these unit test. Also, those 
test cases had passed at my local deployment.

 Export Utility throws ClassNotFoundException
 

 Key: HBASE-6456
 URL: https://issues.apache.org/jira/browse/HBASE-6456
 Project: HBase
  Issue Type: Bug
Reporter: Shengsheng Huang
Assignee: Shengsheng Huang
Priority: Minor
 Attachments: HBASE-6456.patch


 The command I ran is as below:
 bin/hbase org.apache.hadoop.hbase.mapreduce.Driver export t1 ./t1
 And I got the following exceptions:
 attempt_201207261322_0002_m_00_0, Status : FAILED
 Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.util.Bytes
 at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
 at org.apache.hadoop.hbase.client.HTable.init(HTable.java:150)
 at 
 org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:100)
 at 
 org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
 at 
 org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
 at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.au
 ...
 This exception can be resolved by adding hbase common jar into 
 HADOOP_CLASSPATH. But this is an extra step for users and not so convenient. 
 We could add Bytes.class into dependency Jars of the MapReduce job.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6465) Load balancer repeatedly close and open region in the same regionserver.

2012-07-27 Thread shen guanpu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423706#comment-13423706
 ] 

shen guanpu commented on HBASE-6465:


ok thanks
i am not quit catch the rule，sorry！

 Load balancer repeatedly close and open region in the same regionserver.
 

 Key: HBASE-6465
 URL: https://issues.apache.org/jira/browse/HBASE-6465
 Project: HBase
  Issue Type: Bug
  Components: master, regionserver
Affects Versions: 0.94.0
Reporter: shen guanpu

 Through the master and regionserver log,I find load balancer repeatedly
 close and open region in the same regionserver(period in
 hbase.balancer.period ).
 Does this is a bug in load balancer and how can I dig into or avoid this?
 the hbase and hadoop version is
 HBase Version0.94.0, r1332822Hadoop Version0.20.2-cdh3u1,
 rbdafb1dbffd0d5f2fbc6ee022e1c8df6500fd638
 the following is a detail log about the same region
 trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956,
 and it repeats again and again.:
 2012-07-16 00:12:49,843 INFO org.apache.hadoop.hbase.master.HMaster: balance
 hri=trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956.,
 src=192.168.1.2,60020,1342017399608, dest=192.168.1.2,60020,1342002082592
 2012-07-16 00:12:49,843 DEBUG
 org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
 region
 trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956.
 (offlining)
 2012-07-16 00:12:49,843 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
 master:6-0x4384d0a47f40068 Creating unassigned node for
 93caf5147d40f5dd4625e160e1b7e956 in a CLOSING state
 2012-07-16 00:12:49,845 DEBUG
 org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to
 192.168.1.2,60020,1342017399608 for region
 trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956.
 2012-07-16 00:12:50,555 DEBUG
 org.apache.hadoop.hbase.master.AssignmentManager: Handling
 transition=RS_ZK_REGION_CLOSED, server=192.168.1.2,60020,1342017399608,
 region=93caf5147d40f5dd4625e160e1b7e956
 2012-07-16 00:12:50,555 DEBUG
 org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
 event for 93caf5147d40f5dd4625e160e1b7e956
 2012-07-16 00:12:50,555 DEBUG
 org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
 was=trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956.
 state=CLOSED, ts=1342368770556, server=192.168.1.2,60020,1342017399608
 2012-07-16 00:12:50,555 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
 master:6-0x4384d0a47f40068 Creating (or updating) unassigned node for
 93caf5147d40f5dd4625e160e1b7e956 with OFFLINE state
 2012-07-16 00:12:50,558 DEBUG
 org.apache.hadoop.hbase.master.AssignmentManager: Handling
 transition=M_ZK_REGION_OFFLINE, server=10.75.18.34,6,1342017369575,
 region=93caf5147d40f5dd4625e160e1b7e956
 2012-07-16 00:12:50,558 DEBUG
 org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan
 for
 trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956.
 destination server is 192.168.1.2,60020,1342002082592
 2012-07-16 00:12:50,558 DEBUG
 org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan
 for region
 trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956.;
 plan=hri=trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956.,
 src=192.168.1.2,60020,1342017399608, dest=192.168.1.2,60020,1342002082592
 2012-07-16 00:12:50,558 DEBUG
 org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
 trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956.
 to 192.168.1.2,60020,1342002082592
 2012-07-16 00:12:50,574 DEBUG
 org.apache.hadoop.hbase.master.AssignmentManager: Handling
 transition=RS_ZK_REGION_OPENING, server=192.168.1.2,60020,1342017399608,
 region=93caf5147d40f5dd4625e160e1b7e956
 2012-07-16 00:12:50,635 DEBUG
 org.apache.hadoop.hbase.master.AssignmentManager: Handling
 transition=RS_ZK_REGION_OPENING, server=192.168.1.2,60020,1342017399608,
 region=93caf5147d40f5dd4625e160e1b7e956
 2012-07-16 00:12:50,639 DEBUG
 org.apache.hadoop.hbase.master.AssignmentManager: Handling
 transition=RS_ZK_REGION_OPENED, server=192.168.1.2,60020,1342017399608,
 region=93caf5147d40f5dd4625e160e1b7e956
 2012-07-16 00:12:50,639 DEBUG
 org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED
 event for
 trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956.
 from 192.168.1.2,60020,1342017399608; deleting unassigned node
 2012-07-16 00:12:50,640 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
 master:6-0x4384d0a47f40068 Deleting existing unassigned node for
 93caf5147d40f5dd4625e160e1b7e956

[jira] [Assigned] (HBASE-6372) Add scanner batching to Export job

2012-07-27 Thread Shengsheng Huang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shengsheng Huang reassigned HBASE-6372:
---

Assignee: Shengsheng Huang

 Add scanner batching to Export job
 --

 Key: HBASE-6372
 URL: https://issues.apache.org/jira/browse/HBASE-6372
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Affects Versions: 0.96.0, 0.94.2
Reporter: Lars George
Assignee: Shengsheng Huang
Priority: Minor
  Labels: newbie
 Attachments: HBASE-6372.patch


 When a single row is too large for the RS heap then an OOME can take out the 
 entire RS. Setting scanner batching in custom scans helps avoiding this 
 scenario, but for the supplied Export job this is not set.
 Similar to HBASE-3421 we can set the batching to a low number - or if needed 
 make it a command line option.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6461) Killing the HRegionServer and DataNode hosting ROOT can result in a malformed root table.

2012-07-27 Thread ramkrishna.s.vasudevan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423712#comment-13423712
 ] 

ramkrishna.s.vasudevan commented on HBASE-6461:
---

Sorry for pitching in late here. We faced some issues related to 0 length file 
in Hlogs.  As Uma has pointed out in HDFS-3701 or HDFS-3222 we had some patches 
from HDFS and the same some work around was done on HBASE side also to fix this 
issue internally.
I shall discuss with Uma further on this.  Thanks all.

 Killing the HRegionServer and DataNode hosting ROOT can result in a malformed 
 root table.
 -

 Key: HBASE-6461
 URL: https://issues.apache.org/jira/browse/HBASE-6461
 Project: HBase
  Issue Type: Bug
 Environment: hadoop-0.20.2-cdh3u3
 HBase 0.94.1 RC1
Reporter: Elliott Clark
Priority: Critical
 Fix For: 0.94.2


 Spun up a new dfs on hadoop-0.20.2-cdh3u3
 Started hbase
 started running loadtest tool.
 killed rs and dn holding root with killall -9 java on server sv4r27s44 at 
 about 2012-07-25 22:40:00
 After things stabilize Root is in a bad state. Ran hbck and got:
 Exception in thread main 
 org.apache.hadoop.hbase.client.NoServerForRegionException: No server address 
 listed in -ROOT- for region .META.,,1.1028785192 containing row 
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1016)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
 at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232)
 at org.apache.hadoop.hbase.client.HTable.init(HTable.java:172)
 at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:241)
 at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3236)
 hbase(main):001:0 scan '-ROOT-'
 ROW   COLUMN+CELL 
   
 
 12/07/25 22:43:18 INFO security.UserGroupInformation: JAAS Configuration 
 already set up for Hadoop, not re-installing.
  .META.,,1column=info:regioninfo, 
 timestamp=1343255838525, value={NAME = '.META.,,1', STARTKEY = '', ENDKEY 
 = '', ENCODED = 1028785192,}
  .META.,,1column=info:v, 
 timestamp=1343255838525, value=\x00\x00   
  
 1 row(s) in 0.5930 seconds
 Here's the master log: https://gist.github.com/3179194
 I tried the same thing with 0.92.1 and I was able to get into a similar 
 situation, so I don't think this is anything new. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2214) Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly

2012-07-27 Thread Ferdy Galema (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423740#comment-13423740
 ] 

Ferdy Galema commented on HBASE-2214:
-

I'm happy with you or anyone else wrapping this up. (I'm not sure if the 
client/server buffersize mismatch issue deserves an issue on its own or if you 
guys want to solve it in this issue. That's up to you.)

 Do HBASE-1996 -- setting size to return in scan rather than count of rows -- 
 properly
 -

 Key: HBASE-2214
 URL: https://issues.apache.org/jira/browse/HBASE-2214
 Project: HBase
  Issue Type: New Feature
Reporter: stack
 Fix For: 0.96.0, 0.94.2

 Attachments: HBASE-2214-0.94-v2.txt, HBASE-2214-0.94-v3.txt, 
 HBASE-2214-0.94.txt, HBASE-2214-v4.txt, HBASE-2214-v5.txt, HBASE-2214-v6.txt, 
 HBASE-2214-v7.txt, HBASE-2214_with_broken_TestShell.txt


 The notion that you set size rather than row count specifying how many rows a 
 scanner should return in each cycle was raised over in hbase-1966.  Its a 
 good one making hbase regular though the data under it may vary.  
 HBase-1966 was committed but the patch was constrained by the fact that it 
 needed to not change RPC interface.  This issue is about doing hbase-1966 for 
 0.21 in a clean, unconstrained way.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6457) when use bulkload, hundreds of reduce write hfile, maybe create files of the same name, as a result some data loss

2012-07-27 Thread wanbin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

wanbin updated HBASE-6457:
--

Affects Version/s: (was: 0.92.1)
   0.90.0

 when use bulkload, hundreds of reduce write hfile, maybe create files of the 
 same name, as a result some data loss 
 ---

 Key: HBASE-6457
 URL: https://issues.apache.org/jira/browse/HBASE-6457
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 0.90.0
Reporter: wanbin



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6457) when use bulkload, hundreds of reduce write hfile, maybe create files of the same name, as a result some data loss

2012-07-27 Thread wanbin (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423745#comment-13423745
 ] 

wanbin commented on HBASE-6457:
---

sorry 0.92 is fixed this bug.using UUID.
0.90.x has this problem. becauce it using random

 when use bulkload, hundreds of reduce write hfile, maybe create files of the 
 same name, as a result some data loss 
 ---

 Key: HBASE-6457
 URL: https://issues.apache.org/jira/browse/HBASE-6457
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 0.90.0
Reporter: wanbin



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6317) Master clean start up and Partially enabled tables make region assignment inconsistent.

2012-07-27 Thread ramkrishna.s.vasudevan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423758#comment-13423758
 ] 

ramkrishna.s.vasudevan commented on HBASE-6317:
---

Could some one review this?..Infact if HBASE-6272 goes in we have to do some 
rebase here.  Based on the reviews we can prepare update once.
@Stack,@Jimmy
Your comments welcome.



 Master clean start up and Partially enabled tables make region assignment 
 inconsistent.
 ---

 Key: HBASE-6317
 URL: https://issues.apache.org/jira/browse/HBASE-6317
 Project: HBase
  Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: rajeshbabu
 Fix For: 0.92.2, 0.96.0, 0.94.2

 Attachments: HBASE-6317_94.patch, HBASE-6317_94_3.patch


 If we have a  table in partially enabled state (ENABLING) then on HMaster 
 restart we treat it as a clean cluster start up and do a bulk assign.  
 Currently in 0.94 bulk assign will not handle ALREADY_OPENED scenarios and it 
 leads to region assignment problems.  Analysing more on this we found that we 
 have better way to handle these scenarios.
 {code}
 if (false == checkIfRegionBelongsToDisabled(regionInfo)
  false == checkIfRegionsBelongsToEnabling(regionInfo)) {
   synchronized (this.regions) {
 regions.put(regionInfo, regionLocation);
 addToServers(regionLocation, regionInfo);
   }
 {code}
 We dont add to regions map so that enable table handler can handle it.  But 
 as nothing is added to regions map we think it as a clean cluster start up.
 Will come up with a patch tomorrow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6466) Enable multi-thread for Flush

2012-07-27 Thread Zhihong Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Ted Yu updated HBASE-6466:
--

Status: Patch Available  (was: Open)

 Enable multi-thread for Flush
 -

 Key: HBASE-6466
 URL: https://issues.apache.org/jira/browse/HBASE-6466
 Project: HBase
  Issue Type: Improvement
Reporter: chunhui shen
Assignee: chunhui shen
 Attachments: HBASE-6466.patch, HBASE-6466v2.patch


 If the KV is large or Hlog is closed with high-pressure putting, we found 
 memstore is often above the high water mark and block the putting.
 So should we enable multi-thread for Memstore Flush?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6449) Dapper like tracing

2012-07-27 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423769#comment-13423769
 ] 

stack commented on HBASE-6449:
--

@Jonathan Sounds good to me.  Maybe others have opinions otherwise.  Would 
suggest that you keep running commentary on your progress, say here in this 
issue, because there is a bunch of interest in making sure this works for hbase 
so I for one would be up for review and trying it out whats up on github or 
wherever (and as said already, for getting in hooks you need -- perhaps in a 
different issue -- and its ok if you don't nail it the first time... we can 
file new issues to adjust as you learn more what you need).

 Dapper like tracing
 ---

 Key: HBASE-6449
 URL: https://issues.apache.org/jira/browse/HBASE-6449
 Project: HBase
  Issue Type: New Feature
  Components: client, ipc
Affects Versions: 0.96.0
Reporter: Jonathan Leavitt
  Labels: tracing
 Attachments: htrace1.diff, trace.png


 Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to 
 HBase. [Accumulo|http://accumulo.apache.org] added something similar with 
 their cloudtrace package. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6466) Enable multi-thread for Flush

2012-07-27 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423784#comment-13423784
 ] 

Hadoop QA commented on HBASE-6466:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12538126/HBASE-6466v2.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 hadoop2.0.  The patch compiles against the hadoop 2.0 profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

-1 javac.  The applied patch generated 5 javac compiler warnings (more than 
the trunk's current 4 warnings).

-1 findbugs.  The patch appears to introduce 14 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2446//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2446//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2446//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2446//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2446//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2446//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2446//console

This message is automatically generated.

 Enable multi-thread for Flush
 -

 Key: HBASE-6466
 URL: https://issues.apache.org/jira/browse/HBASE-6466
 Project: HBase
  Issue Type: Improvement
Reporter: chunhui shen
Assignee: chunhui shen
 Attachments: HBASE-6466.patch, HBASE-6466v2.patch


 If the KV is large or Hlog is closed with high-pressure putting, we found 
 memstore is often above the high water mark and block the putting.
 So should we enable multi-thread for Memstore Flush?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6466) Enable multi-thread for memstore flush

2012-07-27 Thread Zhihong Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Ted Yu updated HBASE-6466:
--

Summary: Enable multi-thread for memstore flush  (was: Enable multi-thread 
for Flush)

 Enable multi-thread for memstore flush
 --

 Key: HBASE-6466
 URL: https://issues.apache.org/jira/browse/HBASE-6466
 Project: HBase
  Issue Type: Improvement
Reporter: chunhui shen
Assignee: chunhui shen
 Attachments: HBASE-6466.patch, HBASE-6466v2.patch


 If the KV is large or Hlog is closed with high-pressure putting, we found 
 memstore is often above the high water mark and block the putting.
 So should we enable multi-thread for Memstore Flush?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6461) Killing the HRegionServer and DataNode hosting ROOT can result in a malformed root table.

2012-07-27 Thread nkeywal (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423883#comment-13423883
 ] 

nkeywal commented on HBASE-6461:


@Ram
Thanks a lot!

@Lars
bq. I thought HBase neither uses nor needs append (the reason for initial 
using the append branch was that it also had the sync patches).
Yes, the log line is actually confusing, you just need to have someone writing 
a file and someone else reading it.

bq.  Is this only a special case for the WAL when we start to replay the edits, 
when the file is not closed, yet, because the writer died?
I think so. Even if I wonder if we could not have similar stuff for large store 
hfiles + crashes, even if on paper it should be ok.


 Killing the HRegionServer and DataNode hosting ROOT can result in a malformed 
 root table.
 -

 Key: HBASE-6461
 URL: https://issues.apache.org/jira/browse/HBASE-6461
 Project: HBase
  Issue Type: Bug
 Environment: hadoop-0.20.2-cdh3u3
 HBase 0.94.1 RC1
Reporter: Elliott Clark
Priority: Critical
 Fix For: 0.94.2


 Spun up a new dfs on hadoop-0.20.2-cdh3u3
 Started hbase
 started running loadtest tool.
 killed rs and dn holding root with killall -9 java on server sv4r27s44 at 
 about 2012-07-25 22:40:00
 After things stabilize Root is in a bad state. Ran hbck and got:
 Exception in thread main 
 org.apache.hadoop.hbase.client.NoServerForRegionException: No server address 
 listed in -ROOT- for region .META.,,1.1028785192 containing row 
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1016)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
 at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232)
 at org.apache.hadoop.hbase.client.HTable.init(HTable.java:172)
 at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:241)
 at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3236)
 hbase(main):001:0 scan '-ROOT-'
 ROW   COLUMN+CELL 
   
 
 12/07/25 22:43:18 INFO security.UserGroupInformation: JAAS Configuration 
 already set up for Hadoop, not re-installing.
  .META.,,1column=info:regioninfo, 
 timestamp=1343255838525, value={NAME = '.META.,,1', STARTKEY = '', ENDKEY 
 = '', ENCODED = 1028785192,}
  .META.,,1column=info:v, 
 timestamp=1343255838525, value=\x00\x00   
  
 1 row(s) in 0.5930 seconds
 Here's the master log: https://gist.github.com/3179194
 I tried the same thing with 0.92.1 and I was able to get into a similar 
 situation, so I don't think this is anything new. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore flush

2012-07-27 Thread Zhihong Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423922#comment-13423922
 ] 

Zhihong Ted Yu commented on HBASE-6466:
---

Minor comment:
{code}
+  LOG.warn(Memstore is above high water mark and block  + took + ms);
{code}
'is above' - 'was above'
' block ' - ' reclaimMemStoreMemory took '

 Enable multi-thread for memstore flush
 --

 Key: HBASE-6466
 URL: https://issues.apache.org/jira/browse/HBASE-6466
 Project: HBase
  Issue Type: Improvement
Reporter: chunhui shen
Assignee: chunhui shen
 Attachments: HBASE-6466.patch, HBASE-6466v2.patch


 If the KV is large or Hlog is closed with high-pressure putting, we found 
 memstore is often above the high water mark and block the putting.
 So should we enable multi-thread for Memstore Flush?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6272) In-memory region state is inconsistent

2012-07-27 Thread Jimmy Xiang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-6272:
---

   Resolution: Fixed
Fix Version/s: 0.96.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

The failed two tests work fine locally. Actually, all unit tests are green 
locally.

Integrated to trunk. Thank Stack and Ram for reviewing it.

 In-memory region state is inconsistent
 --

 Key: HBASE-6272
 URL: https://issues.apache.org/jira/browse/HBASE-6272
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: 6272-trunk_v6.patch


 AssignmentManger stores region state related information in several places: 
 regionsInTransition, regions (region info to server name map), and servers 
 (server name to region info set map).  However the access to these places is 
 not coordinated properly.  It leads to inconsistent in-memory region state 
 information.  Sometimes, some region could even be offline, and not in 
 transition.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6467) ROOT stuck in assigning forever

2012-07-27 Thread Jimmy Xiang (JIRA) 
Jimmy Xiang created HBASE-6467:
--

 Summary: ROOT stuck in assigning forever
 Key: HBASE-6467
 URL: https://issues.apache.org/jira/browse/HBASE-6467
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.1
Reporter: Jimmy Xiang


After restart a cluster, all region servers checked into master but the master 
stuck in assigning forever.

Master log shows it keeps trying connect to one region server for ROOT table, 
while that region server's log shows it keeps printing out 
NotServingRegionException.

After restart the master, things are ok now.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6467) ROOT stuck in assigning forever

2012-07-27 Thread Jimmy Xiang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-6467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-6467:
---

Attachment: root-region-assignment.png

 ROOT stuck in assigning forever
 ---

 Key: HBASE-6467
 URL: https://issues.apache.org/jira/browse/HBASE-6467
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.1
Reporter: Jimmy Xiang
 Attachments: master.log.gz, regionserver.log.gz, 
 root-region-assignment.png


 After restart a cluster, all region servers checked into master but the 
 master stuck in assigning forever.
 Master log shows it keeps trying connect to one region server for ROOT table, 
 while that region server's log shows it keeps printing out 
 NotServingRegionException.
 After restart the master, things are ok now.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6467) ROOT stuck in assigning forever

2012-07-27 Thread Jimmy Xiang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-6467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-6467:
---

Attachment: regionserver.log.gz
master.log.gz

 ROOT stuck in assigning forever
 ---

 Key: HBASE-6467
 URL: https://issues.apache.org/jira/browse/HBASE-6467
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.1
Reporter: Jimmy Xiang
 Attachments: master.log.gz, regionserver.log.gz, 
 root-region-assignment.png


 After restart a cluster, all region servers checked into master but the 
 master stuck in assigning forever.
 Master log shows it keeps trying connect to one region server for ROOT table, 
 while that region server's log shows it keeps printing out 
 NotServingRegionException.
 After restart the master, things are ok now.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6317) Master clean start up and Partially enabled tables make region assignment inconsistent.

2012-07-27 Thread Jimmy Xiang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424000#comment-13424000
 ] 

Jimmy Xiang commented on HBASE-6317:


HBASE-6272 went in trunk, :)

The inconsistent we are trying to fix here is about the possible 
double-assignment, right?  Any other inconsistency?

If it is just for double-assignment, I think it is better to prevent it from 
AssignmentManager (or the new RegionStates introduced in HBASE-6272).
Whenever a new region is online, if it is assigned to a server different from 
that in RegionStates, we can put the corresponding info to a queue
and make sure it is not still assigned there, without going through zookeeper.

The point is that it's hard to prevent the double-asignment from the source, 
like this issue. The reason is racing and distributed issues.
Of course, it is great if we can prevent it from the source.

Based on a consistent, central place, for example, RegionStates, I think it is 
easier to prevent from some bad things.



 Master clean start up and Partially enabled tables make region assignment 
 inconsistent.
 ---

 Key: HBASE-6317
 URL: https://issues.apache.org/jira/browse/HBASE-6317
 Project: HBase
  Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: rajeshbabu
 Fix For: 0.92.2, 0.96.0, 0.94.2

 Attachments: HBASE-6317_94.patch, HBASE-6317_94_3.patch


 If we have a  table in partially enabled state (ENABLING) then on HMaster 
 restart we treat it as a clean cluster start up and do a bulk assign.  
 Currently in 0.94 bulk assign will not handle ALREADY_OPENED scenarios and it 
 leads to region assignment problems.  Analysing more on this we found that we 
 have better way to handle these scenarios.
 {code}
 if (false == checkIfRegionBelongsToDisabled(regionInfo)
  false == checkIfRegionsBelongsToEnabling(regionInfo)) {
   synchronized (this.regions) {
 regions.put(regionInfo, regionLocation);
 addToServers(regionLocation, regionInfo);
   }
 {code}
 We dont add to regions map so that enable table handler can handle it.  But 
 as nothing is added to regions map we think it as a clean cluster start up.
 Will come up with a patch tomorrow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6272) In-memory region state is inconsistent

2012-07-27 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424003#comment-13424003
 ] 

Hudson commented on HBASE-6272:
---

Integrated in HBase-TRUNK #3177 (See 
[https://builds.apache.org/job/HBase-TRUNK/3177/])
HBASE-6272 In-memory region state is inconsistent (Revision 1366438)

 Result = FAILURE
jxiang : 
Files : 
* 
/hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ClusterStatus.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/UnknownRegionException.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/BulkReOpen.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/NotifiableConcurrentSkipListMap.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionState.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ClosedRegionHandler.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DisableTableHandler.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/EnableTableHandler.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/OnlineRegions.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/Mocking.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java


 In-memory region state is inconsistent
 --

 Key: HBASE-6272
 URL: https://issues.apache.org/jira/browse/HBASE-6272
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: 6272-trunk_v6.patch


 AssignmentManger stores region state related information in several places: 
 regionsInTransition, regions (region info to

[jira] [Commented] (HBASE-6272) In-memory region state is inconsistent

2012-07-27 Thread Jimmy Xiang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424005#comment-13424005
 ] 

Jimmy Xiang commented on HBASE-6272:


The 3 failed tests are ok on my machine.

 In-memory region state is inconsistent
 --

 Key: HBASE-6272
 URL: https://issues.apache.org/jira/browse/HBASE-6272
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: 6272-trunk_v6.patch


 AssignmentManger stores region state related information in several places: 
 regionsInTransition, regions (region info to server name map), and servers 
 (server name to region info set map).  However the access to these places is 
 not coordinated properly.  It leads to inconsistent in-memory region state 
 information.  Sometimes, some region could even be offline, and not in 
 transition.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5547) Don't delete HFiles when in backup mode

2012-07-27 Thread Zhihong Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Ted Yu updated HBASE-5547:
--

Attachment: 5547-addendum-v4.txt

Hopefully this is the last addendum :-)

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-addendum-v4.txt, 5547-v12.txt, 5547-v16.txt, 
 5547.addendum-v3, hbase-5447-v8.patch, hbase-5447-v8.patch, 
 hbase-5547-v9.patch, java_HBASE-5547.addendum, java_HBASE-5547.addendum-v1, 
 java_HBASE-5547.addendum-v2, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6272) In-memory region state is inconsistent

2012-07-27 Thread Zhihong Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424011#comment-13424011
 ] 

Zhihong Ted Yu commented on HBASE-6272:
---

Would there be patches for 0.92 and 0.94 ?

 In-memory region state is inconsistent
 --

 Key: HBASE-6272
 URL: https://issues.apache.org/jira/browse/HBASE-6272
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: 6272-trunk_v6.patch


 AssignmentManger stores region state related information in several places: 
 regionsInTransition, regions (region info to server name map), and servers 
 (server name to region info set map).  However the access to these places is 
 not coordinated properly.  It leads to inconsistent in-memory region state 
 information.  Sometimes, some region could even be offline, and not in 
 transition.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6461) Killing the HRegionServer and DataNode hosting ROOT can result in a malformed root table.

2012-07-27 Thread Lars Hofhansl (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424012#comment-13424012
 ] 

Lars Hofhansl commented on HBASE-6461:
--

I think HFiles are fine, since new HFiles are (or should not) be visible until 
the write succeeded. A compaction write a new file, only when that succeeds are 
the old files deleted.
Flushes are more interesting. If a flush fails half-way because the RS/DN dies, 
the in memory state will be lost and the HLog is the source of truth. But the 
HFile have been 1/2 written. Hmmm.

 Killing the HRegionServer and DataNode hosting ROOT can result in a malformed 
 root table.
 -

 Key: HBASE-6461
 URL: https://issues.apache.org/jira/browse/HBASE-6461
 Project: HBase
  Issue Type: Bug
 Environment: hadoop-0.20.2-cdh3u3
 HBase 0.94.1 RC1
Reporter: Elliott Clark
Priority: Critical
 Fix For: 0.94.2


 Spun up a new dfs on hadoop-0.20.2-cdh3u3
 Started hbase
 started running loadtest tool.
 killed rs and dn holding root with killall -9 java on server sv4r27s44 at 
 about 2012-07-25 22:40:00
 After things stabilize Root is in a bad state. Ran hbck and got:
 Exception in thread main 
 org.apache.hadoop.hbase.client.NoServerForRegionException: No server address 
 listed in -ROOT- for region .META.,,1.1028785192 containing row 
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1016)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
 at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232)
 at org.apache.hadoop.hbase.client.HTable.init(HTable.java:172)
 at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:241)
 at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3236)
 hbase(main):001:0 scan '-ROOT-'
 ROW   COLUMN+CELL 
   
 
 12/07/25 22:43:18 INFO security.UserGroupInformation: JAAS Configuration 
 already set up for Hadoop, not re-installing.
  .META.,,1column=info:regioninfo, 
 timestamp=1343255838525, value={NAME = '.META.,,1', STARTKEY = '', ENDKEY 
 = '', ENCODED = 1028785192,}
  .META.,,1column=info:v, 
 timestamp=1343255838525, value=\x00\x00   
  
 1 row(s) in 0.5930 seconds
 Here's the master log: https://gist.github.com/3179194
 I tried the same thing with 0.92.1 and I was able to get into a similar 
 situation, so I don't think this is anything new. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5705) Introduce Protocol Buffer RPC engine

2012-07-27 Thread Devaraj Das (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424015#comment-13424015
 ] 

Devaraj Das commented on HBASE-5705:


Thanks for the reviews, Stack  Ted.

 Introduce Protocol Buffer RPC engine
 

 Key: HBASE-5705
 URL: https://issues.apache.org/jira/browse/HBASE-5705
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
 Fix For: 0.96.0

 Attachments: 5705-1.patch, 5705-2.1.patch, 5705-2.2.patch


 Introduce Protocol Buffer RPC engine in the RPC core. Protocols that are PB 
 aware can be made to go through this RPC engine. The approach, in my current 
 thinking, would be similar to HADOOP-7773.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6272) In-memory region state is inconsistent

2012-07-27 Thread Jimmy Xiang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424014#comment-13424014
 ] 

Jimmy Xiang commented on HBASE-6272:


I'd like to. Probably after 0.94.1 is released.

 In-memory region state is inconsistent
 --

 Key: HBASE-6272
 URL: https://issues.apache.org/jira/browse/HBASE-6272
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: 6272-trunk_v6.patch


 AssignmentManger stores region state related information in several places: 
 regionsInTransition, regions (region info to server name map), and servers 
 (server name to region info set map).  However the access to these places is 
 not coordinated properly.  It leads to inconsistent in-memory region state 
 information.  Sometimes, some region could even be offline, and not in 
 transition.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover

2012-07-27 Thread Jimmy Xiang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424020#comment-13424020
 ] 

Jimmy Xiang commented on HBASE-6060:


@Rajesh, could you update your patch since HBASE-6272 is in, and put the diff 
on RB?

 Regions's in OPENING state from failed regionservers takes a long time to 
 recover
 -

 Key: HBASE-6060
 URL: https://issues.apache.org/jira/browse/HBASE-6060
 Project: HBase
  Issue Type: Bug
  Components: master, regionserver
Reporter: Enis Soztutar
Assignee: rajeshbabu
 Fix For: 0.96.0, 0.92.3, 0.94.2

 Attachments: 6060-94-v3.patch, 6060-94-v4.patch, 6060-94-v4_1.patch, 
 6060-94-v4_1.patch, 6060-trunk.patch, 6060-trunk.patch, 6060-trunk_2.patch, 
 6060-trunk_3.patch, 6060_alternative_suggestion.txt, 
 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch, 
 6060_suggestion_toassign_rs_wentdown_beforerequest.patch, 
 HBASE-6060-92.patch, HBASE-6060-94.patch, HBASE-6060-trunk_4.patch, 
 HBASE-6060_trunk_5.patch


 we have seen a pattern in tests, that the regions are stuck in OPENING state 
 for a very long time when the region server who is opening the region fails. 
 My understanding of the process: 
  
  - master calls rs to open the region. If rs is offline, a new plan is 
 generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in 
 master memory, zk still shows OFFLINE). See HRegionServer.openRegion(), 
 HMaster.assign()
  - RegionServer, starts opening a region, changes the state in znode. But 
 that znode is not ephemeral. (see ZkAssign)
  - Rs transitions zk node from OFFLINE to OPENING. See 
 OpenRegionHandler.process()
  - rs then opens the region, and changes znode from OPENING to OPENED
  - when rs is killed between OPENING and OPENED states, then zk shows OPENING 
 state, and the master just waits for rs to change the region state, but since 
 rs is down, that wont happen. 
  - There is a AssignmentManager.TimeoutMonitor, which does exactly guard 
 against these kind of conditions. It periodically checks (every 10 sec by 
 default) the regions in transition to see whether they timedout 
 (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min, 
 which explains what you and I are seeing. 
  - ServerShutdownHandler in Master does not reassign regions in OPENING 
 state, although it handles other states. 
 Lowering that threshold from the configuration is one option, but still I 
 think we can do better. 
 Will investigate more. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

2012-07-27 Thread Jesse Yates (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424027#comment-13424027
 ] 

Jesse Yates commented on HBASE-5547:


@Ted how much worse in terms of NN overhead would it be to check if the file 
still exists when not deleted and comment as such? It seems a little excessive 
for just logging, but it would be nice.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-addendum-v4.txt, 5547-v12.txt, 5547-v16.txt, 
 5547.addendum-v3, hbase-5447-v8.patch, hbase-5447-v8.patch, 
 hbase-5547-v9.patch, java_HBASE-5547.addendum, java_HBASE-5547.addendum-v1, 
 java_HBASE-5547.addendum-v2, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore flush

2012-07-27 Thread Jean-Daniel Cryans (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424037#comment-13424037
 ] 

Jean-Daniel Cryans commented on HBASE-6466:
---

How does this relate to HBASE-2832? It's a shame the patch of there was lost 
tho. Jon Gray might still have it somewhere.

Also if this patch doesn't modify the behavior of {{HLog.startCacheFlush}} and 
{{HLog.completeCacheFlush}} WRT the {{cacheFlushLock}} I can't see how it could 
make things any faster. In fact, it's even worse since the other flushing 
thread acquire a write lock just before getting the flush lock (meaning that 
while one thread flushes, the others are blocking the inserts and not flushing).

 Enable multi-thread for memstore flush
 --

 Key: HBASE-6466
 URL: https://issues.apache.org/jira/browse/HBASE-6466
 Project: HBase
  Issue Type: Improvement
Reporter: chunhui shen
Assignee: chunhui shen
 Attachments: HBASE-6466.patch, HBASE-6466v2.patch


 If the KV is large or Hlog is closed with high-pressure putting, we found 
 memstore is often above the high water mark and block the putting.
 So should we enable multi-thread for Memstore Flush?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

2012-07-27 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424051#comment-13424051
 ] 

Hadoop QA commented on HBASE-5547:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12538184/5547-addendum-v4.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 hadoop2.0.  The patch compiles against the hadoop 2.0 profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

-1 javac.  The applied patch generated 5 javac compiler warnings (more than 
the trunk's current 4 warnings).

-1 findbugs.  The patch appears to introduce 5 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2447//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2447//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2447//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2447//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2447//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2447//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2447//console

This message is automatically generated.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-addendum-v4.txt, 5547-v12.txt, 5547-v16.txt, 
 5547.addendum-v3, hbase-5447-v8.patch, hbase-5447-v8.patch, 
 hbase-5547-v9.patch, java_HBASE-5547.addendum, java_HBASE-5547.addendum-v1, 
 java_HBASE-5547.addendum-v2, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

2012-07-27 Thread Zhihong Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424083#comment-13424083
 ] 

Zhihong Ted Yu commented on HBASE-5547:
---

If you want to add an extra check for file existence when fs.delete() returns 
false, that should be done in a sub-task.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-addendum-v4.txt, 5547-v12.txt, 5547-v16.txt, 
 5547.addendum-v3, hbase-5447-v8.patch, hbase-5447-v8.patch, 
 hbase-5547-v9.patch, java_HBASE-5547.addendum, java_HBASE-5547.addendum-v1, 
 java_HBASE-5547.addendum-v2, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

2012-07-27 Thread Zhihong Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424088#comment-13424088
 ] 

Zhihong Ted Yu commented on HBASE-5547:
---

Addendum v4 integrated to trunk.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-addendum-v4.txt, 5547-v12.txt, 5547-v16.txt, 
 5547.addendum-v3, hbase-5447-v8.patch, hbase-5447-v8.patch, 
 hbase-5547-v9.patch, java_HBASE-5547.addendum, java_HBASE-5547.addendum-v1, 
 java_HBASE-5547.addendum-v2, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore flush

2012-07-27 Thread Zhihong Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424121#comment-13424121
 ] 

Zhihong Ted Yu commented on HBASE-6466:
---

{{cacheFlushLock}} is locked by HLog.startCacheFlush() which is called by 
HRegion.internalFlushcache()
In common scenario, this is in the path of region initialization.

MemStoreFlusher, on the other hand, operates on {{MemStoreFlusher.flushQueue}} 
which is a BlockingQueue.

So I think the patch is valid.

 Enable multi-thread for memstore flush
 --

 Key: HBASE-6466
 URL: https://issues.apache.org/jira/browse/HBASE-6466
 Project: HBase
  Issue Type: Improvement
Reporter: chunhui shen
Assignee: chunhui shen
 Attachments: HBASE-6466.patch, HBASE-6466v2.patch


 If the KV is large or Hlog is closed with high-pressure putting, we found 
 memstore is often above the high water mark and block the putting.
 So should we enable multi-thread for Memstore Flush?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore flush

2012-07-27 Thread Jean-Daniel Cryans (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424127#comment-13424127
 ] 

Jean-Daniel Cryans commented on HBASE-6466:
---

internalFlushcache is called by HRegion.flushcache which is called by 
MemStoreFlusher.flushRegion so it is called for every flush.

 Enable multi-thread for memstore flush
 --

 Key: HBASE-6466
 URL: https://issues.apache.org/jira/browse/HBASE-6466
 Project: HBase
  Issue Type: Improvement
Reporter: chunhui shen
Assignee: chunhui shen
 Attachments: HBASE-6466.patch, HBASE-6466v2.patch


 If the KV is large or Hlog is closed with high-pressure putting, we found 
 memstore is often above the high water mark and block the putting.
 So should we enable multi-thread for Memstore Flush?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

2012-07-27 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424138#comment-13424138
 ] 

Hudson commented on HBASE-5547:
---

Integrated in HBase-TRUNK #3178 (See 
[https://builds.apache.org/job/HBase-TRUNK/3178/])
HBASE-5547 Don't delete HFiles when in backup mode, addendum fixes check 
for fs.delete() (Revision 1366510)

 Result = SUCCESS
tedyu : 
Files : 
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java


 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-addendum-v4.txt, 5547-v12.txt, 5547-v16.txt, 
 5547.addendum-v3, hbase-5447-v8.patch, hbase-5447-v8.patch, 
 hbase-5547-v9.patch, java_HBASE-5547.addendum, java_HBASE-5547.addendum-v1, 
 java_HBASE-5547.addendum-v2, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover

2012-07-27 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424142#comment-13424142
 ] 

stack commented on HBASE-6060:
--

@Jimmy I did the last patch.  I think you should review it for 'merit'.  If you 
think it worth it, I'll update to fit hbase-6272.

 Regions's in OPENING state from failed regionservers takes a long time to 
 recover
 -

 Key: HBASE-6060
 URL: https://issues.apache.org/jira/browse/HBASE-6060
 Project: HBase
  Issue Type: Bug
  Components: master, regionserver
Reporter: Enis Soztutar
Assignee: rajeshbabu
 Fix For: 0.96.0, 0.92.3, 0.94.2

 Attachments: 6060-94-v3.patch, 6060-94-v4.patch, 6060-94-v4_1.patch, 
 6060-94-v4_1.patch, 6060-trunk.patch, 6060-trunk.patch, 6060-trunk_2.patch, 
 6060-trunk_3.patch, 6060_alternative_suggestion.txt, 
 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch, 
 6060_suggestion_toassign_rs_wentdown_beforerequest.patch, 
 HBASE-6060-92.patch, HBASE-6060-94.patch, HBASE-6060-trunk_4.patch, 
 HBASE-6060_trunk_5.patch


 we have seen a pattern in tests, that the regions are stuck in OPENING state 
 for a very long time when the region server who is opening the region fails. 
 My understanding of the process: 
  
  - master calls rs to open the region. If rs is offline, a new plan is 
 generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in 
 master memory, zk still shows OFFLINE). See HRegionServer.openRegion(), 
 HMaster.assign()
  - RegionServer, starts opening a region, changes the state in znode. But 
 that znode is not ephemeral. (see ZkAssign)
  - Rs transitions zk node from OFFLINE to OPENING. See 
 OpenRegionHandler.process()
  - rs then opens the region, and changes znode from OPENING to OPENED
  - when rs is killed between OPENING and OPENED states, then zk shows OPENING 
 state, and the master just waits for rs to change the region state, but since 
 rs is down, that wont happen. 
  - There is a AssignmentManager.TimeoutMonitor, which does exactly guard 
 against these kind of conditions. It periodically checks (every 10 sec by 
 default) the regions in transition to see whether they timedout 
 (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min, 
 which explains what you and I are seeing. 
  - ServerShutdownHandler in Master does not reassign regions in OPENING 
 state, although it handles other states. 
 Lowering that threshold from the configuration is one option, but still I 
 think we can do better. 
 Will investigate more. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6272) In-memory region state is inconsistent

2012-07-27 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424143#comment-13424143
 ] 

stack commented on HBASE-6272:
--

I think this change verges on too radical to be backported.  Lets get 0.96 out?

 In-memory region state is inconsistent
 --

 Key: HBASE-6272
 URL: https://issues.apache.org/jira/browse/HBASE-6272
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: 6272-trunk_v6.patch


 AssignmentManger stores region state related information in several places: 
 regionsInTransition, regions (region info to server name map), and servers 
 (server name to region info set map).  However the access to these places is 
 not coordinated properly.  It leads to inconsistent in-memory region state 
 information.  Sometimes, some region could even be offline, and not in 
 transition.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore flush

2012-07-27 Thread Zhihong Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424146#comment-13424146
 ] 

Zhihong Ted Yu commented on HBASE-6466:
---

MemStoreFlusher.flushRegion does the following after region.flushcache() 
returns:
{code}
  boolean shouldSplit = region.checkSplit() != null;
  if (shouldSplit) {
this.server.compactSplitThread.requestSplit(region);
  } else if (shouldCompact) {
server.compactSplitThread.requestCompaction(region, 
Thread.currentThread().getName());
  }
{code}
The above operation is region-wise. The performance boost in tps might have 
come from this.

 Enable multi-thread for memstore flush
 --

 Key: HBASE-6466
 URL: https://issues.apache.org/jira/browse/HBASE-6466
 Project: HBase
  Issue Type: Improvement
Reporter: chunhui shen
Assignee: chunhui shen
 Attachments: HBASE-6466.patch, HBASE-6466v2.patch


 If the KV is large or Hlog is closed with high-pressure putting, we found 
 memstore is often above the high water mark and block the putting.
 So should we enable multi-thread for Memstore Flush?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6468) RowCounter may return incorrect result if column name is specified in command line

2012-07-27 Thread Shrijeet Paliwal (JIRA) 
Shrijeet Paliwal created HBASE-6468:
---

 Summary: RowCounter may return incorrect result if column name is 
specified in command line
 Key: HBASE-6468
 URL: https://issues.apache.org/jira/browse/HBASE-6468
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5
Reporter: Shrijeet Paliwal


The RowCounter use FirstKeyOnlyFilter regardless of whether or not the
command line argument specified a column family (or family:qualifier).
In case when no qualifier was specified as argument, the scan will
give correct result. However in the other case the scan instance may
have been set with columns other than the very first column in the
row, causing scan to get nothing as the FirstKeyOnlyFilter removes
everything else.

https://issues.apache.org/jira/browse/HBASE-6042 is related. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6468) RowCounter may return incorrect result if column name is specified in command line

2012-07-27 Thread Shrijeet Paliwal (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shrijeet Paliwal updated HBASE-6468:


Attachment: 0001-HBASE-6468-RowCounter-may-return-incorrect-result.patch

 RowCounter may return incorrect result if column name is specified in command 
 line
 --

 Key: HBASE-6468
 URL: https://issues.apache.org/jira/browse/HBASE-6468
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5
Reporter: Shrijeet Paliwal
 Attachments: 
 0001-HBASE-6468-RowCounter-may-return-incorrect-result.patch


 The RowCounter use FirstKeyOnlyFilter regardless of whether or not the
 command line argument specified a column family (or family:qualifier).
 In case when no qualifier was specified as argument, the scan will
 give correct result. However in the other case the scan instance may
 have been set with columns other than the very first column in the
 row, causing scan to get nothing as the FirstKeyOnlyFilter removes
 everything else.
 https://issues.apache.org/jira/browse/HBASE-6042 is related. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6468) RowCounter may return incorrect result if column name is specified in command line

2012-07-27 Thread Shrijeet Paliwal (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shrijeet Paliwal updated HBASE-6468:


Status: Patch Available  (was: Open)

 RowCounter may return incorrect result if column name is specified in command 
 line
 --

 Key: HBASE-6468
 URL: https://issues.apache.org/jira/browse/HBASE-6468
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5
Reporter: Shrijeet Paliwal
 Attachments: 
 0001-HBASE-6468-RowCounter-may-return-incorrect-result.patch


 The RowCounter use FirstKeyOnlyFilter regardless of whether or not the
 command line argument specified a column family (or family:qualifier).
 In case when no qualifier was specified as argument, the scan will
 give correct result. However in the other case the scan instance may
 have been set with columns other than the very first column in the
 row, causing scan to get nothing as the FirstKeyOnlyFilter removes
 everything else.
 https://issues.apache.org/jira/browse/HBASE-6042 is related. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6468) RowCounter may return incorrect result if column name is specified in command line

2012-07-27 Thread Zhihong Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424189#comment-13424189
 ] 

Zhihong Ted Yu commented on HBASE-6468:
---

{code}
+if (!scanHasColumns) {
+  scan.setFilter(new FirstKeyOnlyFilter());
+}
{code}
Can we enhance FirstKeyOnlyFilter so that the filter checks for designated 
column before returning ReturnCode.NEXT_ROW ?
The rationale is that we want to speed up scan in case the designated column 
appears in the first half of the row.

Please generate patch for trunk first.

No year is needed in license header.

Thanks

 RowCounter may return incorrect result if column name is specified in command 
 line
 --

 Key: HBASE-6468
 URL: https://issues.apache.org/jira/browse/HBASE-6468
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5
Reporter: Shrijeet Paliwal
 Attachments: 
 0001-HBASE-6468-RowCounter-may-return-incorrect-result.patch


 The RowCounter use FirstKeyOnlyFilter regardless of whether or not the
 command line argument specified a column family (or family:qualifier).
 In case when no qualifier was specified as argument, the scan will
 give correct result. However in the other case the scan instance may
 have been set with columns other than the very first column in the
 row, causing scan to get nothing as the FirstKeyOnlyFilter removes
 everything else.
 https://issues.apache.org/jira/browse/HBASE-6042 is related. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5991) Introduce sequential ZNode based read/write locks

2012-07-27 Thread Alex Feinberg (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424190#comment-13424190
 ] 

Alex Feinberg commented on HBASE-5991:
--

All unit tests for exclusion functionality passing. Few issues remaining in 
handling custom specified timeouts. Will iron out and post a diff early next 
week.

 Introduce sequential ZNode based read/write locks 
 --

 Key: HBASE-5991
 URL: https://issues.apache.org/jira/browse/HBASE-5991
 Project: HBase
  Issue Type: Improvement
Reporter: Alex Feinberg
Assignee: Alex Feinberg

 This is a continuation of HBASE-5494:
 Currently table-level write locks have been implemented using non-sequential 
 ZNodes as part of HBASE-5494 and committed to 89-fb branch. This issue is to 
 track converting the table-level locks to sequential ZNodes and supporting 
 read-write locks, as to solve the issue of preventing schema changes during 
 region splits or merges.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6468) RowCounter may return incorrect result if column name is specified in command line

2012-07-27 Thread Shrijeet Paliwal (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424197#comment-13424197
 ] 

Shrijeet Paliwal commented on HBASE-6468:
-

I do not know enough about filters, so help me here. If we go about modifying 
FirstKeyOnlyFilter, how does one handle a case when rowcounter was invoked with 
something like this : rowcounter table f1:c1 f2:c2 . Create two instances of 
FirstKeyOnlyFilter (one for each pair)? 



 RowCounter may return incorrect result if column name is specified in command 
 line
 --

 Key: HBASE-6468
 URL: https://issues.apache.org/jira/browse/HBASE-6468
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5
Reporter: Shrijeet Paliwal
 Attachments: 
 0001-HBASE-6468-RowCounter-may-return-incorrect-result.patch


 The RowCounter use FirstKeyOnlyFilter regardless of whether or not the
 command line argument specified a column family (or family:qualifier).
 In case when no qualifier was specified as argument, the scan will
 give correct result. However in the other case the scan instance may
 have been set with columns other than the very first column in the
 row, causing scan to get nothing as the FirstKeyOnlyFilter removes
 everything else.
 https://issues.apache.org/jira/browse/HBASE-6042 is related. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6468) RowCounter may return incorrect result if column name is specified in command line

2012-07-27 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424199#comment-13424199
 ] 

Hadoop QA commented on HBASE-6468:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12538216/0001-HBASE-6468-RowCounter-may-return-incorrect-result.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 4 new or modified tests.

-1 patch.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/2448//console

This message is automatically generated.

 RowCounter may return incorrect result if column name is specified in command 
 line
 --

 Key: HBASE-6468
 URL: https://issues.apache.org/jira/browse/HBASE-6468
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5
Reporter: Shrijeet Paliwal
 Attachments: 
 0001-HBASE-6468-RowCounter-may-return-incorrect-result.patch


 The RowCounter use FirstKeyOnlyFilter regardless of whether or not the
 command line argument specified a column family (or family:qualifier).
 In case when no qualifier was specified as argument, the scan will
 give correct result. However in the other case the scan instance may
 have been set with columns other than the very first column in the
 row, causing scan to get nothing as the FirstKeyOnlyFilter removes
 everything else.
 https://issues.apache.org/jira/browse/HBASE-6042 is related. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restart

2012-07-27 Thread Enis Soztutar (JIRA) 
Enis Soztutar created HBASE-6469:


 Summary: Failure on enable/disable table will cause table state in 
zk to be left as enabling/disabling until master is restart
 Key: HBASE-6469
 URL: https://issues.apache.org/jira/browse/HBASE-6469
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.96.0, 0.94.2
Reporter: Enis Soztutar
Assignee: Enis Soztutar


In Enable/DisableTableHandler code, if something goes wrong in handling, the 
table state in zk is left as ENABLING / DISABLING. After that we cannot force 
any more action from the API or CLI, and the only recovery path is restarting 
the master. 

{code}
if (done) {
  // Flip the table to enabled.
  this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
  LOG.info(Table ' + this.tableNameStr
  + ' was successfully enabled. Status: done= + done);
} else {
  LOG.warn(Table ' + this.tableNameStr
  + ' wasn't successfully enabled. Status: done= + done);
}
{code}

Here, if done is false, the table state is not changed. There is also no way to 
set skipTableStateCheck from cli / api. 

We have run into this issue a couple of times before. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6468) RowCounter may return incorrect result if column name is specified in command line

2012-07-27 Thread Zhihong Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Ted Yu updated HBASE-6468:
--

Status: Open  (was: Patch Available)

 RowCounter may return incorrect result if column name is specified in command 
 line
 --

 Key: HBASE-6468
 URL: https://issues.apache.org/jira/browse/HBASE-6468
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5
Reporter: Shrijeet Paliwal
 Attachments: 
 0001-HBASE-6468-RowCounter-may-return-incorrect-result.patch


 The RowCounter use FirstKeyOnlyFilter regardless of whether or not the
 command line argument specified a column family (or family:qualifier).
 In case when no qualifier was specified as argument, the scan will
 give correct result. However in the other case the scan instance may
 have been set with columns other than the very first column in the
 row, causing scan to get nothing as the FirstKeyOnlyFilter removes
 everything else.
 https://issues.apache.org/jira/browse/HBASE-6042 is related. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6468) RowCounter may return incorrect result if column name is specified in command line

2012-07-27 Thread Zhihong Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424205#comment-13424205
 ] 

Zhihong Ted Yu commented on HBASE-6468:
---

My first comment lacks clarity: FirstKeyOnlyFilter would only deal with first 
key.

The use case you listed above adds some complexity. Assuming the table is very 
wide, there is reason for user to do that.

We can create new Filter whose ctor takes the columns user specifies. Its 
filterKeyValue() method would look for the given columns in KeyValue parameter. 
Once there is a match for any one of the columns, we can return 
ReturnCode.NEXT_ROW for remaining KeyValues in the row.

Hope this helps.

 RowCounter may return incorrect result if column name is specified in command 
 line
 --

 Key: HBASE-6468
 URL: https://issues.apache.org/jira/browse/HBASE-6468
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5
Reporter: Shrijeet Paliwal
 Attachments: 
 0001-HBASE-6468-RowCounter-may-return-incorrect-result.patch


 The RowCounter use FirstKeyOnlyFilter regardless of whether or not the
 command line argument specified a column family (or family:qualifier).
 In case when no qualifier was specified as argument, the scan will
 give correct result. However in the other case the scan instance may
 have been set with columns other than the very first column in the
 row, causing scan to get nothing as the FirstKeyOnlyFilter removes
 everything else.
 https://issues.apache.org/jira/browse/HBASE-6042 is related. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6272) In-memory region state is inconsistent

2012-07-27 Thread Jimmy Xiang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424208#comment-13424208
 ] 

Jimmy Xiang commented on HBASE-6272:


Sure.  I am fine with that.

 In-memory region state is inconsistent
 --

 Key: HBASE-6272
 URL: https://issues.apache.org/jira/browse/HBASE-6272
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: 6272-trunk_v6.patch


 AssignmentManger stores region state related information in several places: 
 regionsInTransition, regions (region info to server name map), and servers 
 (server name to region info set map).  However the access to these places is 
 not coordinated properly.  It leads to inconsistent in-memory region state 
 information.  Sometimes, some region could even be offline, and not in 
 transition.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover

2012-07-27 Thread Jimmy Xiang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424209#comment-13424209
 ] 

Jimmy Xiang commented on HBASE-6060:


@Stack, no problem, doesn't have to.

 Regions's in OPENING state from failed regionservers takes a long time to 
 recover
 -

 Key: HBASE-6060
 URL: https://issues.apache.org/jira/browse/HBASE-6060
 Project: HBase
  Issue Type: Bug
  Components: master, regionserver
Reporter: Enis Soztutar
Assignee: rajeshbabu
 Fix For: 0.96.0, 0.92.3, 0.94.2

 Attachments: 6060-94-v3.patch, 6060-94-v4.patch, 6060-94-v4_1.patch, 
 6060-94-v4_1.patch, 6060-trunk.patch, 6060-trunk.patch, 6060-trunk_2.patch, 
 6060-trunk_3.patch, 6060_alternative_suggestion.txt, 
 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch, 
 6060_suggestion_toassign_rs_wentdown_beforerequest.patch, 
 HBASE-6060-92.patch, HBASE-6060-94.patch, HBASE-6060-trunk_4.patch, 
 HBASE-6060_trunk_5.patch


 we have seen a pattern in tests, that the regions are stuck in OPENING state 
 for a very long time when the region server who is opening the region fails. 
 My understanding of the process: 
  
  - master calls rs to open the region. If rs is offline, a new plan is 
 generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in 
 master memory, zk still shows OFFLINE). See HRegionServer.openRegion(), 
 HMaster.assign()
  - RegionServer, starts opening a region, changes the state in znode. But 
 that znode is not ephemeral. (see ZkAssign)
  - Rs transitions zk node from OFFLINE to OPENING. See 
 OpenRegionHandler.process()
  - rs then opens the region, and changes znode from OPENING to OPENED
  - when rs is killed between OPENING and OPENED states, then zk shows OPENING 
 state, and the master just waits for rs to change the region state, but since 
 rs is down, that wont happen. 
  - There is a AssignmentManager.TimeoutMonitor, which does exactly guard 
 against these kind of conditions. It periodically checks (every 10 sec by 
 default) the regions in transition to see whether they timedout 
 (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min, 
 which explains what you and I are seeing. 
  - ServerShutdownHandler in Master does not reassign regions in OPENING 
 state, although it handles other states. 
 Lowering that threshold from the configuration is one option, but still I 
 think we can do better. 
 Will investigate more. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore flush

2012-07-27 Thread Jean-Daniel Cryans (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424210#comment-13424210
 ] 

Jean-Daniel Cryans commented on HBASE-6466:
---

bq. The above operation is region-wise. The performance boost in tps might have 
come from this.

Those are just queueing requests to the compact split thread, I don't think I 
follow you.

 Enable multi-thread for memstore flush
 --

 Key: HBASE-6466
 URL: https://issues.apache.org/jira/browse/HBASE-6466
 Project: HBase
  Issue Type: Improvement
Reporter: chunhui shen
Assignee: chunhui shen
 Attachments: HBASE-6466.patch, HBASE-6466v2.patch


 If the KV is large or Hlog is closed with high-pressure putting, we found 
 memstore is often above the high water mark and block the putting.
 So should we enable multi-thread for Memstore Flush?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore flush

2012-07-27 Thread Zhihong Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424213#comment-13424213
 ] 

Zhihong Ted Yu commented on HBASE-6466:
---

CompactSplitThread.requestCompaction(final HRegion r, final Store s,
  final String why, int priority) has this call (line 225): 
{code}
  pool.execute(cr);
{code}
which leads to CompactionRequest line 252:
{code}
boolean completed = r.compact(this);
{code}

 Enable multi-thread for memstore flush
 --

 Key: HBASE-6466
 URL: https://issues.apache.org/jira/browse/HBASE-6466
 Project: HBase
  Issue Type: Improvement
Reporter: chunhui shen
Assignee: chunhui shen
 Attachments: HBASE-6466.patch, HBASE-6466v2.patch


 If the KV is large or Hlog is closed with high-pressure putting, we found 
 memstore is often above the high water mark and block the putting.
 So should we enable multi-thread for Memstore Flush?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (HBASE-6466) Enable multi-thread for memstore flush

2012-07-27 Thread Zhihong Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424213#comment-13424213
 ] 

Zhihong Ted Yu edited comment on HBASE-6466 at 7/27/12 11:21 PM:
-

CompactSplitThread.requestCompaction(final HRegion r, final Store s,
  final String why, int priority) has this call (line 225): 
{code}
  pool.execute(cr);
{code}
which leads to CompactionRequest line 252:
{code}
boolean completed = r.compact(this);
{code}
This doesn't explain the performance gain :-)

  was (Author: zhi...@ebaysf.com):
CompactSplitThread.requestCompaction(final HRegion r, final Store s,
  final String why, int priority) has this call (line 225): 
{code}
  pool.execute(cr);
{code}
which leads to CompactionRequest line 252:
{code}
boolean completed = r.compact(this);
{code}
  
 Enable multi-thread for memstore flush
 --

 Key: HBASE-6466
 URL: https://issues.apache.org/jira/browse/HBASE-6466
 Project: HBase
  Issue Type: Improvement
Reporter: chunhui shen
Assignee: chunhui shen
 Attachments: HBASE-6466.patch, HBASE-6466v2.patch


 If the KV is large or Hlog is closed with high-pressure putting, we found 
 memstore is often above the high water mark and block the putting.
 So should we enable multi-thread for Memstore Flush?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6272) In-memory region state is inconsistent

2012-07-27 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424217#comment-13424217
 ] 

Hudson commented on HBASE-6272:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #113 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/113/])
HBASE-6272 In-memory region state is inconsistent (Revision 1366438)

 Result = FAILURE
jxiang : 
Files : 
* 
/hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ClusterStatus.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/UnknownRegionException.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/BulkReOpen.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/NotifiableConcurrentSkipListMap.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionState.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ClosedRegionHandler.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DisableTableHandler.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/EnableTableHandler.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/OnlineRegions.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/Mocking.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java


 In-memory region state is inconsistent
 --

 Key: HBASE-6272
 URL: https://issues.apache.org/jira/browse/HBASE-6272
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: 6272-trunk_v6.patch


 AssignmentManger stores region state related information in several places:

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

2012-07-27 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424218#comment-13424218
 ] 

Hudson commented on HBASE-5547:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #113 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/113/])
HBASE-5547 Don't delete HFiles when in backup mode, addendum fixes check 
for fs.delete() (Revision 1366510)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java


 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-addendum-v4.txt, 5547-v12.txt, 5547-v16.txt, 
 5547.addendum-v3, hbase-5447-v8.patch, hbase-5447-v8.patch, 
 hbase-5547-v9.patch, java_HBASE-5547.addendum, java_HBASE-5547.addendum-v1, 
 java_HBASE-5547.addendum-v2, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore flush

2012-07-27 Thread chunhui shen (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424233#comment-13424233
 ] 

chunhui shen commented on HBASE-6466:
-

Sorry, I forgot removing lock race in HLog when making patch for trunk,
I'll update the patch later, outting now...

Sorry, it's my mistake...

 Enable multi-thread for memstore flush
 --

 Key: HBASE-6466
 URL: https://issues.apache.org/jira/browse/HBASE-6466
 Project: HBase
  Issue Type: Improvement
Reporter: chunhui shen
Assignee: chunhui shen
 Attachments: HBASE-6466.patch, HBASE-6466v2.patch


 If the KV is large or Hlog is closed with high-pressure putting, we found 
 memstore is often above the high water mark and block the putting.
 So should we enable multi-thread for Memstore Flush?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6427) Pluggable compaction policies via coprocessors

2012-07-27 Thread Lars Hofhansl (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-6427:
-

Attachment: 6427-v4.txt

This has all the functionality I want.
The new TTL test is subject to races if the test env is *really* slow.

Will play with EnvironmentEdgeManager to control time for these test to avoid 
these races.

Please review the other parts of the code.
Is there some better refactoring I can do to StoreScanner in order to make this 
a bit easier on anybody who wants override some functionality?

 Pluggable compaction policies via coprocessors
 --

 Key: HBASE-6427
 URL: https://issues.apache.org/jira/browse/HBASE-6427
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt, 
 6427-v3.txt, 6427-v4.txt


 When implementing higher level stores on top of HBase it is necessary to 
 allow dynamic control over how long KVs must be kept around.
 Semi-static config options for ColumnFamilies (# of version or TTL) is not 
 sufficient.
 This can be done with a few additional coprocessor hooks, or by makeing 
 Store.ScanInfo pluggable.
 Was:
 The simplest way to achieve this is to have a pluggable class to determine 
 the smallestReadpoint for Region. That way outside code can control what KVs 
 to retain.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6468) RowCounter may return incorrect result if column name is specified in command line

2012-07-27 Thread Shrijeet Paliwal (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shrijeet Paliwal updated HBASE-6468:


Attachment: 0002-HBASE-6468-RowCounter-may-return-incorrect-result.patch

This patch is again off 0.90 branch. Take a look if it looks fine I will 
generate a final patch off of trunk. I have not taken care of 'year' in 
copyright. Will do that in final patch. 

 RowCounter may return incorrect result if column name is specified in command 
 line
 --

 Key: HBASE-6468
 URL: https://issues.apache.org/jira/browse/HBASE-6468
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5
Reporter: Shrijeet Paliwal
 Attachments: 
 0001-HBASE-6468-RowCounter-may-return-incorrect-result.patch, 
 0002-HBASE-6468-RowCounter-may-return-incorrect-result.patch


 The RowCounter use FirstKeyOnlyFilter regardless of whether or not the
 command line argument specified a column family (or family:qualifier).
 In case when no qualifier was specified as argument, the scan will
 give correct result. However in the other case the scan instance may
 have been set with columns other than the very first column in the
 row, causing scan to get nothing as the FirstKeyOnlyFilter removes
 everything else.
 https://issues.apache.org/jira/browse/HBASE-6042 is related. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restart

2012-07-27 Thread Lars Hofhansl (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-6469:
-

Fix Version/s: 0.94.2

 Failure on enable/disable table will cause table state in zk to be left as 
 enabling/disabling until master is restart
 -

 Key: HBASE-6469
 URL: https://issues.apache.org/jira/browse/HBASE-6469
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.96.0, 0.94.2
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 0.94.2


 In Enable/DisableTableHandler code, if something goes wrong in handling, the 
 table state in zk is left as ENABLING / DISABLING. After that we cannot force 
 any more action from the API or CLI, and the only recovery path is restarting 
 the master. 
 {code}
 if (done) {
   // Flip the table to enabled.
   this.assignmentManager.getZKTable().setEnabledTable(
 this.tableNameStr);
   LOG.info(Table ' + this.tableNameStr
   + ' was successfully enabled. Status: done= + done);
 } else {
   LOG.warn(Table ' + this.tableNameStr
   + ' wasn't successfully enabled. Status: done= + done);
 }
 {code}
 Here, if done is false, the table state is not changed. There is also no way 
 to set skipTableStateCheck from cli / api. 
 We have run into this issue a couple of times before. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6468) RowCounter may return incorrect result if column name is specified in command line

2012-07-27 Thread Zhihong Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424242#comment-13424242
 ] 

Zhihong Ted Yu commented on HBASE-6468:
---

Patch v2 is on the right track.

There were some spelling mistakes.

There was unnecessary formatting in RowCounter.java

It's a pity that mapred/RowCounter.java doesn't use FirstKeyOnlyFilter. It's up 
to you whether you want to cover that class in this JIRA.

TestFirstKeyMatchingQualifiersFilter should be named 
TestFirstKeyValueMatchingQualifiersFilter.

 RowCounter may return incorrect result if column name is specified in command 
 line
 --

 Key: HBASE-6468
 URL: https://issues.apache.org/jira/browse/HBASE-6468
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5
Reporter: Shrijeet Paliwal
 Attachments: 
 0001-HBASE-6468-RowCounter-may-return-incorrect-result.patch, 
 0002-HBASE-6468-RowCounter-may-return-incorrect-result.patch


 The RowCounter use FirstKeyOnlyFilter regardless of whether or not the
 command line argument specified a column family (or family:qualifier).
 In case when no qualifier was specified as argument, the scan will
 give correct result. However in the other case the scan instance may
 have been set with columns other than the very first column in the
 row, causing scan to get nothing as the FirstKeyOnlyFilter removes
 everything else.
 https://issues.apache.org/jira/browse/HBASE-6042 is related. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6468) RowCounter may return incorrect result if column name is specified in command line

2012-07-27 Thread Zhihong Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424244#comment-13424244
 ] 

Zhihong Ted Yu commented on HBASE-6468:
---

{code}
+Listbyte[] qualifiers = new ArrayListbyte[]();
{code}
Would suggest replacing the List with a Set because the same qualifier might be 
entered more than once.

 RowCounter may return incorrect result if column name is specified in command 
 line
 --

 Key: HBASE-6468
 URL: https://issues.apache.org/jira/browse/HBASE-6468
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5
Reporter: Shrijeet Paliwal
 Attachments: 
 0001-HBASE-6468-RowCounter-may-return-incorrect-result.patch, 
 0002-HBASE-6468-RowCounter-may-return-incorrect-result.patch


 The RowCounter use FirstKeyOnlyFilter regardless of whether or not the
 command line argument specified a column family (or family:qualifier).
 In case when no qualifier was specified as argument, the scan will
 give correct result. However in the other case the scan instance may
 have been set with columns other than the very first column in the
 row, causing scan to get nothing as the FirstKeyOnlyFilter removes
 everything else.
 https://issues.apache.org/jira/browse/HBASE-6042 is related. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6427) Pluggable compaction policies via coprocessors

2012-07-27 Thread Lars Hofhansl (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-6427:
-

Attachment: 6427-v5.txt

- used EnvironmentEdge in TestCoprocessorScanPolicy to avoid races.
- added ScanType which was missing in v4

 Pluggable compaction policies via coprocessors
 --

 Key: HBASE-6427
 URL: https://issues.apache.org/jira/browse/HBASE-6427
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt, 
 6427-v3.txt, 6427-v4.txt, 6427-v5.txt


 When implementing higher level stores on top of HBase it is necessary to 
 allow dynamic control over how long KVs must be kept around.
 Semi-static config options for ColumnFamilies (# of version or TTL) is not 
 sufficient.
 This can be done with a few additional coprocessor hooks, or by makeing 
 Store.ScanInfo pluggable.
 Was:
 The simplest way to achieve this is to have a pluggable class to determine 
 the smallestReadpoint for Region. That way outside code can control what KVs 
 to retain.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors

2012-07-27 Thread Zhihong Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424270#comment-13424270
 ] 

Zhihong Ted Yu commented on HBASE-6427:
---

Some new files need license header.
{code}
+public class ScanPolicyCoprocessor extends BaseRegionObserver {
{code}
This class is an observer. Suggest renaming the class.
Annotations for audience and stability should be added.

For RegionCoprocessorHost.preCompact():
{code}
+  scanner = ((RegionObserver) env.getInstance()).preCompact(ctx, 
store, scanners,
+  scanType, earliestPutTs);
{code}
If there're multiple RegionObserver's, it seems only the final returned scanner 
would be returned. Is this intentional ?
Similar observation for preStoreScannerOpen() and preFlush()

 Pluggable compaction policies via coprocessors
 --

 Key: HBASE-6427
 URL: https://issues.apache.org/jira/browse/HBASE-6427
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt, 
 6427-v3.txt, 6427-v4.txt, 6427-v5.txt


 When implementing higher level stores on top of HBase it is necessary to 
 allow dynamic control over how long KVs must be kept around.
 Semi-static config options for ColumnFamilies (# of version or TTL) is not 
 sufficient.
 This can be done with a few additional coprocessor hooks, or by makeing 
 Store.ScanInfo pluggable.
 Was:
 The simplest way to achieve this is to have a pluggable class to determine 
 the smallestReadpoint for Region. That way outside code can control what KVs 
 to retain.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors

2012-07-27 Thread Lars Hofhansl (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424273#comment-13424273
 ] 

Lars Hofhansl commented on HBASE-6427:
--

Thanks Ted. You are right on all counts. Need to think about multiple 
coprocessors a bit more.

 Pluggable compaction policies via coprocessors
 --

 Key: HBASE-6427
 URL: https://issues.apache.org/jira/browse/HBASE-6427
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt, 
 6427-v3.txt, 6427-v4.txt, 6427-v5.txt


 When implementing higher level stores on top of HBase it is necessary to 
 allow dynamic control over how long KVs must be kept around.
 Semi-static config options for ColumnFamilies (# of version or TTL) is not 
 sufficient.
 This can be done with a few additional coprocessor hooks, or by makeing 
 Store.ScanInfo pluggable.
 Was:
 The simplest way to achieve this is to have a pluggable class to determine 
 the smallestReadpoint for Region. That way outside code can control what KVs 
 to retain.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors

2012-07-27 Thread Lars Hofhansl (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424275#comment-13424275
 ] 

Lars Hofhansl commented on HBASE-6427:
--

The current preCompact/preScannerOpen/other hooks handle this by passing the 
scanner from the previous coprocessor to the next one, so that each coprocessor 
has all the information needed.
I could do something similar here, although it would quickly get inscrutable 
for an implemented of these hooks; but I cannot think of anything better.


 Pluggable compaction policies via coprocessors
 --

 Key: HBASE-6427
 URL: https://issues.apache.org/jira/browse/HBASE-6427
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt, 
 6427-v3.txt, 6427-v4.txt, 6427-v5.txt


 When implementing higher level stores on top of HBase it is necessary to 
 allow dynamic control over how long KVs must be kept around.
 Semi-static config options for ColumnFamilies (# of version or TTL) is not 
 sufficient.
 This can be done with a few additional coprocessor hooks, or by makeing 
 Store.ScanInfo pluggable.
 Was:
 The simplest way to achieve this is to have a pluggable class to determine 
 the smallestReadpoint for Region. That way outside code can control what KVs 
 to retain.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira