date:20120119


 [ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chunhui shen updated HBASE-5179:


Attachment: 5179-90v12.patch

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 
 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 
 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 
 5179-v4.txt, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v5.patch, 
 hbase-5179v6.patch, hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss


 [ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chunhui shen updated HBASE-5179:


Attachment: hbase-5179v12.patch

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 
 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 
 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 
 5179-v4.txt, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v12.patch, 
 hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, 
 hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss

2012-01-19 Thread chunhui shen (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188983#comment-13188983
 ] 

chunhui shen commented on HBASE-5179:
-

@Jinchao

I think v12 has consider the above cases, could you help to test this patch in 
your cluster ?

I have tried to write an unit test case, but the simulated kill would delete 
EphemeralNode on zk when Regionserver Thread exit, so the above case is hard to 
appear through unit test case.

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 
 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 
 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 
 5179-v4.txt, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v12.patch, 
 hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, 
 hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5216) [book] book.xml - added more in Arch when to use hbase section


[ 
https://issues.apache.org/jira/browse/HBASE-5216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189003#comment-13189003
 ] 

Hudson commented on HBASE-5216:
---

Integrated in HBase-TRUNK #2639 (See 
[https://builds.apache.org/job/HBase-TRUNK/2639/])
hbase-5216 book.xml - added detail on Arch when to use hbase section


 [book] book.xml - added more in Arch when to use hbase section
 

 Key: HBASE-5216
 URL: https://issues.apache.org/jira/browse/HBASE-5216
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: book_hbase_5216.xml.patch


 book.xml - Arch chapter, added an additional paragraph in the when to use 
 hbase section.  This is based on a lengthy dist-list conversation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5153) Add retry logic in HConnectionImplementation#resetZooKeeperTrackers


[ 
https://issues.apache.org/jira/browse/HBASE-5153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189005#comment-13189005
 ] 

Hudson commented on HBASE-5153:
---

Integrated in HBase-TRUNK #2639 (See 
[https://builds.apache.org/job/HBase-TRUNK/2639/])
HBASE-5153 revert due to failed Jenkins builds

tedyu : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClosedConnectionException.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnection.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/MasterSchemaChangeTracker.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/SchemaChangeTracker.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java


 Add retry logic in HConnectionImplementation#resetZooKeeperTrackers
 ---

 Key: HBASE-5153
 URL: https://issues.apache.org/jira/browse/HBASE-5153
 Project: HBase
  Issue Type: Bug
  Components: client
Affects Versions: 0.90.4
Reporter: Jieshan Bean
Assignee: Jieshan Bean
 Fix For: 0.94.0, 0.92.1, 0.90.6

 Attachments: 5153-92.txt, 5153-trunk.txt, 5153-trunk.txt, 
 HBASE-5153-V2.patch, HBASE-5153-V3.patch, HBASE-5153-V4-90.patch, 
 HBASE-5153-V5-90.patch, HBASE-5153-V6-90-minorchange.patch, 
 HBASE-5153-V6-90.txt, HBASE-5153-trunk-v2.patch, HBASE-5153-trunk.patch, 
 HBASE-5153.patch, TestResults-hbase5153.out


 HBASE-4893 is related to this issue. In that issue, we know, if multi-threads 
 share a same connection, once this connection got abort in one thread, the 
 other threads will got a 
 HConnectionManager$HConnectionImplementation@18fb1f7 closed exception.
 It solve the problem of stale connection can't removed. But the orignal 
 HTable instance cann't be continue to use. The connection in HTable should be 
 recreated.
 Actually, there's two aproach to solve this:
 1. In user code, once catch an IOE, close connection and re-create HTable 
 instance. We can use this as a workaround.
 2. In HBase Client side, catch this exception, and re-create connection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5223) TestMetaReaderEditor is missing call to CatalogTracker.stop()


[ 
https://issues.apache.org/jira/browse/HBASE-5223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189006#comment-13189006
 ] 

Hudson commented on HBASE-5223:
---

Integrated in HBase-TRUNK #2639 (See 
[https://builds.apache.org/job/HBase-TRUNK/2639/])
HBASE-5223  TestMetaReaderEditor is missing call to CatalogTracker.stop()

tedyu : 
Files : 
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java


 TestMetaReaderEditor is missing call to CatalogTracker.stop()
 -

 Key: HBASE-5223
 URL: https://issues.apache.org/jira/browse/HBASE-5223
 Project: HBase
  Issue Type: Test
Reporter: Zhihong Yu
 Fix For: 0.94.0, 0.92.1

 Attachments: 5223.txt


 I noticed that TestMetaReaderEditor hung on 0.92 Jenkins builds - see 
 https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/249/console
 It turns out that CatalogTracker.stop() is missing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5203) Group atomic put/delete operation into a single WALEdit to handle region server failures.


[ 
https://issues.apache.org/jira/browse/HBASE-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189004#comment-13189004
 ] 

Hudson commented on HBASE-5203:
---

Integrated in HBase-TRUNK #2639 (See 
[https://builds.apache.org/job/HBase-TRUNK/2639/])
HBASE-5203 Group atomic put/delete operation into a single WALEdit to 
handle region server failures. (Lars H)

larsh : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Delete.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java


 Group atomic put/delete operation into a single WALEdit to handle region 
 server failures.
 -

 Key: HBASE-5203
 URL: https://issues.apache.org/jira/browse/HBASE-5203
 Project: HBase
  Issue Type: Sub-task
  Components: client, coprocessors, regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5203-v3.txt, 5203.txt


 HBASE-3584 does not not provide fully atomic operation in case of region 
 server failures (see explanation there).
 What should happen is that either (1) all edits are applied via a single 
 WALEdit, or (2) the WALEdits are applied in async mode and then sync'ed 
 together.
 For #1 it is not clear whether it is advisable to manage multiple *different* 
 operations (Put/Delete) via a single WAL edit. A quick check reveals that WAL 
 replay on region startup would work, but that replication would need to be 
 adapted. The refactoring needed would be non-trivial.
 #2 Might actually not work, as another operation could request sync'ing a 
 later edit and hence flush these entries out as well.
 Addendum:
 The attached patch implements #1 and fixes replication to be able to deal 
 with different operations being grouped in one WALEdit.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5226) [book] troubleshooting.xml - add client slowdown when calling admin APIs issue to Client section


[ 
https://issues.apache.org/jira/browse/HBASE-5226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189007#comment-13189007
 ] 

Hudson commented on HBASE-5226:
---

Integrated in HBase-TRUNK #2639 (See 
[https://builds.apache.org/job/HBase-TRUNK/2639/])
hbase-5226.  troubleshooting.xml, client section:  adding reference to 
client slowdown due to admin API calls.


 [book] troubleshooting.xml - add client slowdown when calling admin APIs 
 issue to Client section
 

 Key: HBASE-5226
 URL: https://issues.apache.org/jira/browse/HBASE-5226
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: troubleshooting_hbase_5226.xml.patch


 troubleshooting.xml - adding client slowdown when calling admin APIs issue 
 (HBASE-5073) to the Client section.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5208) Allow setting Scan start/stop row individually in TableInputFormat


[ 
https://issues.apache.org/jira/browse/HBASE-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189011#comment-13189011
 ] 

Hudson commented on HBASE-5208:
---

Integrated in HBase-TRUNK #2639 (See 
[https://builds.apache.org/job/HBase-TRUNK/2639/])
HBASE-5208 Allow setting Scan start/stop row individually in 
TableInputFormat

tedyu : 
Files : 
* /hbase/trunk/CHANGES.txt
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormat.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScan.java


 Allow setting Scan start/stop row individually in TableInputFormat
 --

 Key: HBASE-5208
 URL: https://issues.apache.org/jira/browse/HBASE-5208
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Reporter: Nicholas Telford
Priority: Minor
 Attachments: HBASE-5208-001.txt, HBASE-5208-002.txt, 
 HBASE-5208-003.txt, HBASE-5208-004.txt


 Currently, TableInputFormat initializes a serialized Scan from 
 hbase.mapreduce.scan. Alternatively, it will instantiate a new Scan using 
 properties defined in hbase.mapreduce.scan.*. However, of these properties 
 the start row and stop row (arguably the most pertinent) are missing.
 TableInputFormat should permit the specification of a start/stop row as with 
 the other fields using a new pair of properties: 
 hbase.mapreduce.scan.row.start and hbase.mapreduce.scan.row.end
 The primary use-case for this is to permit Oozie and other job management 
 tools that can't call TableMapReduceUtil.initTableMapperJob() to operate on a 
 contiguous subset of rows.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5220) [book] troubleshooting.xml - adding information about 'zkcli' to Troubleshooting chapter


[ 
https://issues.apache.org/jira/browse/HBASE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189013#comment-13189013
 ] 

Hudson commented on HBASE-5220:
---

Integrated in HBase-TRUNK #2639 (See 
[https://builds.apache.org/job/HBase-TRUNK/2639/])
hbase-5220 troubleshooting.xml - zkcli info


 [book] troubleshooting.xml - adding information about 'zkcli' to 
 Troubleshooting chapter
 

 Key: HBASE-5220
 URL: https://issues.apache.org/jira/browse/HBASE-5220
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: troubleshooting_hbase_5220.xml.patch


 troubleshooting.xml
 * added entry in 'builtin' tools section on zkcli
 * added link in trouble.zookeeper.general to the above entry
 This utility came up on the dist-list recently.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5201) Utilize TThreadedSelectorServer and remove redundant code in ThriftServer and HRegionThriftServer


[ 
https://issues.apache.org/jira/browse/HBASE-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189008#comment-13189008
 ] 

Hudson commented on HBASE-5201:
---

Integrated in HBase-TRUNK #2639 (See 
[https://builds.apache.org/job/HBase-TRUNK/2639/])
HBASE-5201 Add new files
HBASE-5201 Utilize TThreadedSelectorServer and remove redundant code in 
ThriftServer and HRegionThriftServer (Scott Chen)

tedyu : 
Files : 
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/HThreadedSelectorServerArgs.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java

tedyu : 
Files : 
* /hbase/trunk/pom.xml
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServer.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServerCmdLine.java


 Utilize TThreadedSelectorServer and remove redundant code in ThriftServer and 
 HRegionThriftServer
 -

 Key: HBASE-5201
 URL: https://issues.apache.org/jira/browse/HBASE-5201
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Scott Chen
Assignee: Scott Chen
 Fix For: 0.94.0

 Attachments: HBASE-5201-v2.txt, HBASE-5201-v3.txt, HBASE-5201-v4.txt, 
 HBASE-5201.txt


 TThreadedSelectorServer is good for RPC-heavy situation because IO are not 
 limited to one CPU. See
 https://issues.apache.org/jira/browse/Thrift-1167
 I am porting the related classes form thrift trunk (it is not there in 
 thrift-0.7.0).
 There are lots of repeat codes in ThriftServer and HRegionThriftServer.
 These codes are now moved to a Runnable called ThriftServerRunner.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5176) AssignmentManager#getRegion: logging nit adds a redundant '+'


[ 
https://issues.apache.org/jira/browse/HBASE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189014#comment-13189014
 ] 

Hudson commented on HBASE-5176:
---

Integrated in HBase-TRUNK #2639 (See 
[https://builds.apache.org/job/HBase-TRUNK/2639/])
HBASE-5176  AssignmentManager#getRegion: logging nit  adds a redundant '+' 
(Karthik K)

tedyu : 
Files : 
* /hbase/trunk/CHANGES.txt
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java


 AssignmentManager#getRegion: logging nit  adds a redundant '+'  
 

 Key: HBASE-5176
 URL: https://issues.apache.org/jira/browse/HBASE-5176
 Project: HBase
  Issue Type: Bug
 Environment: hadoop 1.0.0 , zk 3.4.2 , hbase 0.92.0 rc3
Reporter: Karthik K
Assignee: Karthik K
Priority: Minor
 Fix For: 0.94.0, 0.92.1

 Attachments: HBASE-5176.patch


 From the logs of HMaster: 
 2012-01-10 17:28:24,370 DEBUG 
 org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for 
 -ROOT-,,0.70236052 destination server is + localhost,60020,1326242475275
 Was the '+' intended to be there , as part of some token for log verification 
 or just being redundant , w.r.t the following string append ? 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-1621) merge tool should work on online cluster, but disabled table


[ 
https://issues.apache.org/jira/browse/HBASE-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189010#comment-13189010
 ] 

Hudson commented on HBASE-1621:
---

Integrated in HBase-TRUNK #2639 (See 
[https://builds.apache.org/job/HBase-TRUNK/2639/])
[book] book.xml, ops_mgt.xml  additional clarification that compaction does 
not do region merging
also, added link to script in HBASE-1621 in Ops Mgt chapter.

dmeil : 
Files : 
* /hbase/trunk/src/docbkx/book.xml
* /hbase/trunk/src/docbkx/ops_mgt.xml


 merge tool should work on online cluster, but disabled table
 

 Key: HBASE-1621
 URL: https://issues.apache.org/jira/browse/HBASE-1621
 Project: HBase
  Issue Type: Bug
Reporter: ryan rawson
Assignee: stack
 Fix For: 0.94.0

 Attachments: 1621-trunk.txt, HBASE-1621-v2.patch, HBASE-1621.patch, 
 hbase-onlinemerge.patch, online_merge.rb


 taking down the entire cluster to merge 2 regions is a pain, i dont see why 
 the table or regions specifically couldnt be taken offline, then merged then 
 brought back up.
 this might need a new API to the regionservers so they can take direction 
 from not just the master.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5218) [book] book.xml - added Arch/Hfile, added link to HFile v2 info in appendix


[ 
https://issues.apache.org/jira/browse/HBASE-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189012#comment-13189012
 ] 

Hudson commented on HBASE-5218:
---

Integrated in HBase-TRUNK #2639 (See 
[https://builds.apache.org/job/HBase-TRUNK/2639/])
hbase-5218 book.xml - Arch/HFile, added link to HFile v2 info in appendix 
from this section.


 [book] book.xml - added Arch/Hfile, added link to HFile v2 info in appendix
 ---

 Key: HBASE-5218
 URL: https://issues.apache.org/jira/browse/HBASE-5218
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Trivial
 Attachments: book_hbase_5218.xml.patch


 Stack asked me to do this in December:  added link in Arch/HFile to the HFile 
 v2 information in the appendix.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss

[
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189028#comment-13189028
]

Hadoop QA commented on HBASE-5179:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511103/hbase-5179v12.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.

-1 javadoc. The javadoc tool appears to have generated -143 warning
messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 82 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/810//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/810//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/810//console

This message is automatically generated.

Concurrent processing of processFaileOver and ServerShutdownHandler may cause
region to be assigned before log splitting is completed, causing data loss

Key: HBASE-5179
URL: https://issues.apache.org/jira/browse/HBASE-5179
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.92.0, 0.94.0, 0.90.6

Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch,
5179-90v12.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch,
5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch,
5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt,
5179-v4.txt, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v12.patch,
hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch,
hbase-5179v8.patch, hbase-5179v9.patch

If master's processing its failover and ServerShutdownHandler's processing
happen concurrently, it may appear following case.
1.master completed splitLogAfterStartup()
2.RegionserverA restarts, and ServerShutdownHandler is processing.
3.master starts to rebuildUserRegions, and RegionserverA is considered as
dead server.
4.master starts to assign regions of RegionserverA because it is a dead
server by step3.
However, when doing step4(assigning region), ServerShutdownHandler may be
doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss

[
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189039#comment-13189039
]

Hadoop QA commented on HBASE-5179:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511103/hbase-5179v12.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 javadoc. The javadoc tool appears to have generated -143 warning
messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 82 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.io.hfile.TestLruBlockCache

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/811//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/811//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/811//console

This message is automatically generated.

Concurrent processing of processFaileOver and ServerShutdownHandler may cause
region to be assigned before log splitting is completed, causing data loss

[jira] [Commented] (HBASE-2315) BookKeeper for write-ahead logging

2012-01-19 Thread Flavio Junqueira (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189041#comment-13189041
 ] 

Flavio Junqueira commented on HBASE-2315:
-

Hi Zhihong, Thanks for the bringing this up, and yes, I'm still interested in 
providing an updated patch. I'll work on one.

 BookKeeper for write-ahead logging
 --

 Key: HBASE-2315
 URL: https://issues.apache.org/jira/browse/HBASE-2315
 Project: HBase
  Issue Type: New Feature
  Components: regionserver
Reporter: Flavio Junqueira
 Attachments: HBASE-2315.patch, bookkeeperOverview.pdf, 
 zookeeper-dev-bookkeeper.jar


 BookKeeper, a contrib of the ZooKeeper project, is a fault tolerant and high 
 throughput write-ahead logging service. This issue provides an implementation 
 of write-ahead logging for hbase using BookKeeper. Apart from expected 
 throughput improvements, BookKeeper also has stronger durability guarantees 
 compared to the implementation currently used by hbase.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-2315) BookKeeper for write-ahead logging


 [ 
https://issues.apache.org/jira/browse/HBASE-2315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-2315:
--

Comment: was deleted

(was: I am out of office in an all-day business meeting today and tomorrow,
I will not be checking email. If this is urgent then please call my
cell phone (or send an SMS), otherwise I will reply to your email when
I get back.

Thanks for your patience,

-- amr
)

 BookKeeper for write-ahead logging
 --

 Key: HBASE-2315
 URL: https://issues.apache.org/jira/browse/HBASE-2315
 Project: HBase
  Issue Type: New Feature
  Components: regionserver
Reporter: Flavio Junqueira
 Attachments: HBASE-2315.patch, bookkeeperOverview.pdf, 
 zookeeper-dev-bookkeeper.jar


 BookKeeper, a contrib of the ZooKeeper project, is a fault tolerant and high 
 throughput write-ahead logging service. This issue provides an implementation 
 of write-ahead logging for hbase using BookKeeper. Apart from expected 
 throughput improvements, BookKeeper also has stronger durability guarantees 
 compared to the implementation currently used by hbase.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss


[ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189057#comment-13189057
 ] 

Zhihong Yu commented on HBASE-5179:
---

@Chunhui:
If you can upload v11 to review board, publish it then upload v12, we would be 
able to see the differences. 

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 
 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 
 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 
 5179-v4.txt, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v12.patch, 
 hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, 
 hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss


[ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189066#comment-13189066
 ] 

Zhihong Yu commented on HBASE-5179:
---

https://reviews.apache.org/r/3545 was created with patch v11, then updated with 
patch v12.

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 
 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 
 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 
 5179-v4.txt, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v12.patch, 
 hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, 
 hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss


[ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189140#comment-13189140
 ] 

gaojinchao commented on HBASE-5179:
---

@Chunhui
In 90v12, Maybe below code(metaServerInfo/rootServerInfo) has some 
nullexception?

  // MetaServer is may being processed as dead server. Before assign meta,
  // we need to wait until its log is splitted.
  waitUntilNoLogDir(metaServerInfo.getServerName());
  if (!this.serverManager.isDeadMetaServerInProgress()) {
this.assignmentManager.assignMeta();
  }

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 
 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 
 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 
 5179-v4.txt, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v12.patch, 
 hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, 
 hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss


[ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189148#comment-13189148
 ] 

gaojinchao commented on HBASE-5179:
---

@chunhui
I only verify the branch90 version. I don't have the cluster for branch92(that 
needs a few day, we are planing to setup some test cluster.)

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 
 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 
 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 
 5179-v4.txt, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v12.patch, 
 hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, 
 hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss

2012-01-19 Thread gaojinchao (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

gaojinchao updated HBASE-5179:
--

Attachment: Errorlog

when region server checkin, I killed meta/root region. I found splitHlog is 
after meta was assigned. So I think may has another problem. tomorrow continue 
to dig .

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 
 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 
 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 
 5179-v4.txt, Errorlog, hbase-5179.patch, hbase-5179v10.patch, 
 hbase-5179v12.patch, hbase-5179v5.patch, hbase-5179v6.patch, 
 hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss

[
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189193#comment-13189193
]

Hadoop QA commented on HBASE-5179:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511134/Errorlog
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/812//console

This message is automatically generated.

Concurrent processing of processFaileOver and ServerShutdownHandler may cause
region to be assigned before log splitting is completed, causing data loss

Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch,
5179-90v12.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch,
5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch,
5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt,
5179-v4.txt, Errorlog, hbase-5179.patch, hbase-5179v10.patch,
hbase-5179v12.patch, hbase-5179v5.patch, hbase-5179v6.patch,
hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch

[jira] [Updated] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss

[
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Zhihong Yu updated HBASE-5179:
--

Comment: was deleted

(was: -1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511134/Errorlog
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/812//console

This message is automatically generated.)

Concurrent processing of processFaileOver and ServerShutdownHandler may cause
region to be assigned before log splitting is completed, causing data loss

Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch,
5179-90v12.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch,
5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch,
5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt,
5179-v4.txt, Errorlog, hbase-5179.patch, hbase-5179v10.patch,
hbase-5179v12.patch, hbase-5179v5.patch, hbase-5179v6.patch,
hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss


[ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189203#comment-13189203
 ] 

Zhihong Yu commented on HBASE-5179:
---

@Jinchao:
So C3S31 was carrying .META., right ?
I need to add DEBUG log into waitUntilNoLogDir() which would give us:
1. serverName
2. log directory path
3. how long waitUntilNoLogDir() waited.

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 
 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 
 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 
 5179-v4.txt, Errorlog, hbase-5179.patch, hbase-5179v10.patch, 
 hbase-5179v12.patch, hbase-5179v5.patch, hbase-5179v6.patch, 
 hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5231) Backport HBASE-3373 to 0.90

2012-01-19 Thread Zhihong Yu (Created) (JIRA)

Backport HBASE-3373 to 0.90
---

 Key: HBASE-5231
 URL: https://issues.apache.org/jira/browse/HBASE-5231
 Project: HBase
  Issue Type: Improvement
Reporter: Zhihong Yu


This JIRA backports per-table load balancing to 0.90

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5228) [REST] Rip out transform feature

2012-01-19 Thread Ben West (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189220#comment-13189220
 ] 

Ben West commented on HBASE-5228:
-

Thanks Andrew for this patch.

For others who might run into this: running a jstack on our REST server 
revealed that virtually every thread was hung waiting on a table descriptor:

{code}
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:885)
- locked 0x00070218e7d0 (a 
org.apache.hadoop.hbase.ipc.HBaseClient$Call)
at 
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
at $Proxy61.getHTableDescriptors(Unknown Source)
at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHTableDescriptor(HConnectionManager.java:1871)
at 
org.apache.hadoop.hbase.client.HTable.getTableDescriptor(HTable.java:404)
at 
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.getTableDescriptor(HTablePool.java:344)
at 
org.apache.hadoop.hbase.rest.RowResultGenerator.init(RowResultGenerator.java:64)
at 
org.apache.hadoop.hbase.rest.ResultGenerator.fromRowSpec(ResultGenerator.java:35)
at org.apache.hadoop.hbase.rest.RowResource.get(RowResource.java:86)
{code}

So if your REST server is running slowly on an affected version, I guess try 
running a jstack and see if you see this behavior.

 [REST] Rip out transform feature
 --

 Key: HBASE-5228
 URL: https://issues.apache.org/jira/browse/HBASE-5228
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0, 0.94.0, 0.90.5
Reporter: Andrew Purtell
Assignee: Andrew Purtell
 Fix For: 0.94.0, 0.92.1, 0.90.6

 Attachments: HBASE-5228-0.92.patch, HBASE-5228-trunk.patch


 The 'transform' feature, where REST can be instructed, via a table attribute, 
 to apply a transformation (e.g. base64 encoding or decoding) to a (sub)set of 
 column values before serving them up to a client or storing them into HBase, 
 was added some time ago at the request of Jack Levin. I have since come to 
 regret it, it was not a well thought out feature:
   - This is really an application concern.
   - It adds significant overhead to request processing: Periodically a 
 HBaseAdmin is used to retrieve the table descriptor, in order to scan through 
 table attributes for transformation directives. 
 I think it is best to rip it out, its a real problem area, and REST should be 
 no more concerned about data formats than the Java API. 
 I doubt anyone uses this, not even Jack. Will need to follow up with him to 
 confirm.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5228) [REST] Rip out transform feature


[ 
https://issues.apache.org/jira/browse/HBASE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189222#comment-13189222
 ] 

Zhihong Yu commented on HBASE-5228:
---

@Ben:
The above strace was captured without Andy's patch, right ?

Do you want to try out his patch and see if REST server gets faster ?

Thanks

 [REST] Rip out transform feature
 --

 Key: HBASE-5228
 URL: https://issues.apache.org/jira/browse/HBASE-5228
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0, 0.94.0, 0.90.5
Reporter: Andrew Purtell
Assignee: Andrew Purtell
 Fix For: 0.94.0, 0.92.1, 0.90.6

 Attachments: HBASE-5228-0.92.patch, HBASE-5228-trunk.patch


 The 'transform' feature, where REST can be instructed, via a table attribute, 
 to apply a transformation (e.g. base64 encoding or decoding) to a (sub)set of 
 column values before serving them up to a client or storing them into HBase, 
 was added some time ago at the request of Jack Levin. I have since come to 
 regret it, it was not a well thought out feature:
   - This is really an application concern.
   - It adds significant overhead to request processing: Periodically a 
 HBaseAdmin is used to retrieve the table descriptor, in order to scan through 
 table attributes for transformation directives. 
 I think it is best to rip it out, its a real problem area, and REST should be 
 no more concerned about data formats than the Java API. 
 I doubt anyone uses this, not even Jack. Will need to follow up with him to 
 confirm.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5231) Backport HBASE-3373 (per-table load balancing) to 0.90

2012-01-19 Thread Todd Lipcon (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HBASE-5231:
---

Summary: Backport HBASE-3373 (per-table load balancing) to 0.90  (was: 
Backport HBASE-3373 to 0.90)

 Backport HBASE-3373 (per-table load balancing) to 0.90
 --

 Key: HBASE-5231
 URL: https://issues.apache.org/jira/browse/HBASE-5231
 Project: HBase
  Issue Type: Improvement
Reporter: Zhihong Yu

 This JIRA backports per-table load balancing to 0.90

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5229) Support atomic region operations

2012-01-19 Thread Lars Hofhansl (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189240#comment-13189240
]

Lars Hofhansl commented on HBASE-5229:
--

That argument pretty much sinks this approach for me.

At the same time we should not put a full transactional API into HBase, but
rather provide enough building blocks so that an outside client could implement
transactions. I do not see how we can do that without exposing knowledge about
some internals such as regions.

Another approach is to give more control over which set of rows can participate
in a transaction.
Right now that is all KVs with the same row-key (internally we achieve by
collocating all those KVs, but that is an implementation detail).
What if we allow a prefix of the row key instead? We can even formalize that,
and give the row key some internal (optional) structure, which allows the
application to specific transaction groups.

Support atomic region operations

Key: HBASE-5229
URL: https://issues.apache.org/jira/browse/HBASE-5229
Project: HBase
Issue Type: New Feature
Components: client, regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Fix For: 0.94.0

Attachments: 5229.txt

As discussed (at length) on the dev mailing list with the HBASE-3584 and
HBASE-5203 committed, supporting atomic cross row transactions within a
region becomes simple.
I am aware of the hesitation about the usefulness of this feature, but we
have to start somewhere.
Let's use this jira for discussion, I'll attach a patch (with tests)
momentarily to make this concrete.

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss


[ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189239#comment-13189239
 ] 

Zhihong Yu commented on HBASE-5179:
---

@Stack:
Thanks for your review.
Looks like we still have some work to really solve this issue. After that we 
can align naming to your liking.

w.r.t. local log splitting, to my knowledge there is no priority in distributed 
log splitting. The rationale is to make log splitting of .META. server quick.

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 
 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 
 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 
 5179-v4.txt, Errorlog, hbase-5179.patch, hbase-5179v10.patch, 
 hbase-5179v12.patch, hbase-5179v5.patch, hbase-5179v6.patch, 
 hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5231) Backport HBASE-3373 (per-table load balancing) to 0.90

2012-01-19 Thread Jean-Daniel Cryans (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189244#comment-13189244
 ] 

Jean-Daniel Cryans commented on HBASE-5231:
---

I'm -1, let's keep 0.90 stable and not add new behaviors in late releases. Also 
HBASE-3373 wasn't even applied to 0.92 and has no assignee.

 Backport HBASE-3373 (per-table load balancing) to 0.90
 --

 Key: HBASE-5231
 URL: https://issues.apache.org/jira/browse/HBASE-5231
 Project: HBase
  Issue Type: Improvement
Reporter: Zhihong Yu

 This JIRA backports per-table load balancing to 0.90

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss


[ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189245#comment-13189245
 ] 

stack commented on HBASE-5179:
--

So your thinking is that we'd go local in case a distributed log splitting is 
already going on?  That would work for the case where say ten servers went down 
around same time and the tenth queued had meta on it.  But then, what if there 
is no distributed log splitting going on?  Master-only log splitting would be 
slower than distributed splitting?

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 
 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 
 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 
 5179-v4.txt, Errorlog, hbase-5179.patch, hbase-5179v10.patch, 
 hbase-5179v12.patch, hbase-5179v5.patch, hbase-5179v6.patch, 
 hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5231) Backport HBASE-3373 (per-table load balancing) to 0.92


 [ 
https://issues.apache.org/jira/browse/HBASE-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5231:
--

Summary: Backport HBASE-3373 (per-table load balancing) to 0.92  (was: 
Backport HBASE-3373 (per-table load balancing) to 0.90)

Thanks for the reminder, J-D.

This should go to 0.92 first.

 Backport HBASE-3373 (per-table load balancing) to 0.92
 --

 Key: HBASE-5231
 URL: https://issues.apache.org/jira/browse/HBASE-5231
 Project: HBase
  Issue Type: Improvement
Reporter: Zhihong Yu

 This JIRA backports per-table load balancing to 0.90

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-3373) Allow regions to be load-balanced by table

2012-01-19 Thread Zhihong Yu (Assigned) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu reassigned HBASE-3373:
-

Assignee: Zhihong Yu

 Allow regions to be load-balanced by table
 --

 Key: HBASE-3373
 URL: https://issues.apache.org/jira/browse/HBASE-3373
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 0.20.6
Reporter: Ted Yu
Assignee: Zhihong Yu
 Fix For: 0.94.0

 Attachments: 3373.txt, HbaseBalancerTest2.java


 From our experience, cluster can be well balanced and yet, one table's 
 regions may be badly concentrated on few region servers.
 For example, one table has 839 regions (380 regions at time of table 
 creation) out of which 202 are on one server.
 It would be desirable for load balancer to distribute regions for specified 
 tables evenly across the cluster. Each of such tables has number of regions 
 many times the cluster size.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5231) Backport HBASE-3373 (per-table load balancing) to 0.92

2012-01-19 Thread Jean-Daniel Cryans (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189255#comment-13189255
 ] 

Jean-Daniel Cryans commented on HBASE-5231:
---

I'm -0 on 0.92.

 Backport HBASE-3373 (per-table load balancing) to 0.92
 --

 Key: HBASE-5231
 URL: https://issues.apache.org/jira/browse/HBASE-5231
 Project: HBase
  Issue Type: Improvement
Reporter: Zhihong Yu

 This JIRA backports per-table load balancing to 0.90

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5228) [REST] Rip out transform feature

2012-01-19 Thread Andrew Purtell (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189256#comment-13189256
]

Andrew Purtell commented on HBASE-5228:
---

@Ted: I observed the blocking on this code path eliminated in runs under
jprofiler after the patch was applied.

@Ben: It would be good to confirm this on your side.

[REST] Rip out transform feature
--

Key: HBASE-5228
URL: https://issues.apache.org/jira/browse/HBASE-5228
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.94.0, 0.90.5
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 0.94.0, 0.92.1, 0.90.6

Attachments: HBASE-5228-0.92.patch, HBASE-5228-trunk.patch

The 'transform' feature, where REST can be instructed, via a table attribute,
to apply a transformation (e.g. base64 encoding or decoding) to a (sub)set of
column values before serving them up to a client or storing them into HBase,
was added some time ago at the request of Jack Levin. I have since come to
regret it, it was not a well thought out feature:
- This is really an application concern.
- It adds significant overhead to request processing: Periodically a
HBaseAdmin is used to retrieve the table descriptor, in order to scan through
table attributes for transformation directives.
I think it is best to rip it out, its a real problem area, and REST should be
no more concerned about data formats than the Java API.
I doubt anyone uses this, not even Jack. Will need to follow up with him to
confirm.

[jira] [Commented] (HBASE-5228) [REST] Rip out transform feature

2012-01-19 Thread Ben West (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189267#comment-13189267
]

Ben West commented on HBASE-5228:
-

@Zhihong, Andrew: yes, that stack trace was prior to Andrew's patch. Installing
the patch now, will update with results.

[REST] Rip out transform feature
--

Attachments: HBASE-5228-0.92.patch, HBASE-5228-trunk.patch

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss


[ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189270#comment-13189270
 ] 

Zhihong Yu commented on HBASE-5179:
---

Maybe you're suggesting adding functionality to distributed log splitting which 
allows us to know the length of queue.
That would be a new feature which would go in a separate JIRA.

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 
 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 
 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 
 5179-v4.txt, Errorlog, hbase-5179.patch, hbase-5179v10.patch, 
 hbase-5179v12.patch, hbase-5179v5.patch, hbase-5179v6.patch, 
 hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5229) Support atomic region operations


[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189272#comment-13189272
 ] 

stack commented on HBASE-5229:
--

bq. Another approach is to give more control over which set of rows can 
participate in a transaction.

If we did row prefix instead, it'd have to be an input to the table splitting 
function so we didn't split in the middle of a transactions row set.  Doesn't 
sound hard.  Would be part of table schema.

 Support atomic region operations
 

 Key: HBASE-5229
 URL: https://issues.apache.org/jira/browse/HBASE-5229
 Project: HBase
  Issue Type: New Feature
  Components: client, regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5229.txt


 As discussed (at length) on the dev mailing list with the HBASE-3584 and 
 HBASE-5203 committed, supporting atomic cross row transactions within a 
 region becomes simple.
 I am aware of the hesitation about the usefulness of this feature, but we 
 have to start somewhere.
 Let's use this jira for discussion, I'll attach a patch (with tests) 
 momentarily to make this concrete.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5229) Support atomic region operations

2012-01-19 Thread Todd Lipcon (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189273#comment-13189273
 ] 

Todd Lipcon commented on HBASE-5229:


bq. What if we allow a prefix of the row key instead?
That's essentially the approach that both DynamoDB and Oracle's NoSQL db take. 
But, the way I see it, we already have that -- the row key is the prefix of 
the row key, and the column key is the rest of the row key. What would an 
extra element of key structure give us?

 Support atomic region operations
 

 Key: HBASE-5229
 URL: https://issues.apache.org/jira/browse/HBASE-5229
 Project: HBase
  Issue Type: New Feature
  Components: client, regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5229.txt


 As discussed (at length) on the dev mailing list with the HBASE-3584 and 
 HBASE-5203 committed, supporting atomic cross row transactions within a 
 region becomes simple.
 I am aware of the hesitation about the usefulness of this feature, but we 
 have to start somewhere.
 Let's use this jira for discussion, I'll attach a patch (with tests) 
 momentarily to make this concrete.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss


[ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189277#comment-13189277
 ] 

Zhihong Yu commented on HBASE-5179:
---

{code}
2012-01-19 11:10:35,847 INFO 
org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer ephemeral 
node deleted, processing expiration [C3S31,20020,1326987900492]
2012-01-19 11:10:35,850 DEBUG org.apache.hadoop.hbase.master.ServerManager: 
Added=C3S31,20020,1326987900492 to dead servers, submitted shutdown handler to 
be executed, root=true, meta=false
2012-01-19 11:10:35,851 INFO 
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Splitting logs 
for C3S31,20020,1326987900492
...
2012-01-19 11:12:37,003 INFO 
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log 
hdfs://C3S31:9000/hbase/.logs/C3S31,20020,1326987900492/C3S31%3A20020.1326987901100
 to hdfs://C3S31:9000/hbase/.oldlogs/C3S31%3A20020.1326987901100
2012-01-19 11:12:37,004 INFO 
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: hlog file splitting 
completed in 1137 ms for hdfs://C3S31:9000/hbase/.logs/C3S31,20020,1326987900492
2012-01-19 11:12:37,004 INFO 
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Server 
C3S31,20020,1326987900492 was carrying ROOT. Trying to assign.
...
2012-01-19 11:12:37,382 INFO org.apache.hadoop.hbase.master.HMaster: .META. 
assigned=2, rit=false, location=C3S32:20020
2012-01-19 11:12:37,382 INFO org.apache.hadoop.hbase.master.HMaster: Master 
startup proceeding: master failover
2012-01-19 11:12:37,383 WARN org.apache.hadoop.hbase.master.AssignmentManager: 
Overwriting 1028785192 on serverName=C3S32,20020,1326987400439, 
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
2012-01-19 11:12:37,384 WARN org.apache.hadoop.hbase.master.AssignmentManager: 
Overwriting 70236052 on serverName=C3S32,20020,1326987400439, load=(requests=0, 
regions=0, usedHeap=0, maxHeap=0)
2012-01-19 11:12:37,393 INFO 
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Reassigning 0 
region(s) that C3S31,20020,1326987900492 was carrying (skipping 0 regions(s) 
that are already in transition)
2012-01-19 11:12:37,393 INFO 
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished 
processing of shutdown of C3S31,20020,1326987900492
{code}
From above log, we can see that .META. was assigned to C3S32 and log splitting 
was for C3S31.

So was there data loss ?


 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 
 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 
 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 
 5179-v4.txt, Errorlog, hbase-5179.patch, hbase-5179v10.patch, 
 hbase-5179v12.patch, hbase-5179v5.patch, hbase-5179v6.patch, 
 hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5231) Backport HBASE-3373 (per-table load balancing) to 0.92


[ 
https://issues.apache.org/jira/browse/HBASE-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189279#comment-13189279
 ] 

stack commented on HBASE-5231:
--

Does this feature have to go into 0.92?  Can we do bug fixes in 0.92 only?

 Backport HBASE-3373 (per-table load balancing) to 0.92
 --

 Key: HBASE-5231
 URL: https://issues.apache.org/jira/browse/HBASE-5231
 Project: HBase
  Issue Type: Improvement
Reporter: Zhihong Yu

 This JIRA backports per-table load balancing to 0.90

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5231) Backport HBASE-3373 (per-table load balancing) to 0.92


[ 
https://issues.apache.org/jira/browse/HBASE-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189283#comment-13189283
 ] 

Zhihong Yu commented on HBASE-5231:
---

From thread on lily-discuss entitle: 'Lily Performance':

 * and then, check that the regions of each table are assigned round-robin, 
thus that each region server has a similar amount of regions of each table.
More on this at http://docs.outerthought.org/lily-docs-current/g6/562-lily.html

If user has to use script to achieve the above, we should implement it in core.

 Backport HBASE-3373 (per-table load balancing) to 0.92
 --

 Key: HBASE-5231
 URL: https://issues.apache.org/jira/browse/HBASE-5231
 Project: HBase
  Issue Type: Improvement
Reporter: Zhihong Yu

 This JIRA backports per-table load balancing to 0.90

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5231) Backport HBASE-3373 (per-table load balancing) to 0.92

2012-01-19 Thread Jean-Daniel Cryans (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189294#comment-13189294
 ] 

Jean-Daniel Cryans commented on HBASE-5231:
---

bq. If user has to use script to achieve the above, we should implement it in 
core.

I think you already did.

 Backport HBASE-3373 (per-table load balancing) to 0.92
 --

 Key: HBASE-5231
 URL: https://issues.apache.org/jira/browse/HBASE-5231
 Project: HBase
  Issue Type: Improvement
Reporter: Zhihong Yu

 This JIRA backports per-table load balancing to 0.90

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss


[ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189326#comment-13189326
 ] 

Zhihong Yu commented on HBASE-5179:
---

In patch v10:
{code}
+this.fileSystemManager.splitLog(metaServerInfo.getServerName());
+this.serverManager.expireServer(metaServerInfo);
{code}
In latest patch, fileSystemManager.splitLog() is gone.

I think what might have happened was that waitUntilNoLogDir() returned too soon 
because the log splitting was carried out in ShutdownHandler which barely got a 
chance of doing the split.

Patch v13 adds DEBUG logging in waitUntilNoLogDir() so that we know whether the 
log dir exists upon its entry and how long waitUntilNoLogDir() takes before 
returning.

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 
 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 
 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 
 5179-v4.txt, Errorlog, hbase-5179.patch, hbase-5179v10.patch, 
 hbase-5179v12.patch, hbase-5179v5.patch, hbase-5179v6.patch, 
 hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss


[ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189333#comment-13189333
 ] 

stack commented on HBASE-5179:
--

Why would there be dataloss?  Was C3S32 carrying .META.?  Even if it was, it 
looks like the log split before .META. assign (and this issue is about 
concurrent master failover going on anyways?)

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 
 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 
 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 
 5179-v4.txt, Errorlog, hbase-5179.patch, hbase-5179v10.patch, 
 hbase-5179v12.patch, hbase-5179v5.patch, hbase-5179v6.patch, 
 hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss


[ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189338#comment-13189338
 ] 

Zhihong Yu commented on HBASE-5179:
---

If my assumption @ 19/Jan/12 19:38 is true, we should re-introduce 
this.fileSystemManager.splitLog()

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v13.txt, 5179-90v2.patch, 5179-90v3.patch, 
 5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 
 5179-90v8.patch, 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 
 5179-v3.txt, 5179-v4.txt, Errorlog, hbase-5179.patch, hbase-5179v10.patch, 
 hbase-5179v12.patch, hbase-5179v5.patch, hbase-5179v6.patch, 
 hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss


 [ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5179:
--

Attachment: 5179-90v13.txt

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v13.txt, 5179-90v2.patch, 5179-90v3.patch, 
 5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 
 5179-90v8.patch, 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 
 5179-v3.txt, 5179-v4.txt, Errorlog, hbase-5179.patch, hbase-5179v10.patch, 
 hbase-5179v12.patch, hbase-5179v5.patch, hbase-5179v6.patch, 
 hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5232) hbase-daemons.sh cannot be used do launch zookeeper independently from HBase

2012-01-19 Thread nkeywal (Created) (JIRA)

hbase-daemons.sh cannot be used do launch zookeeper independently from HBase


 Key: HBASE-5232
 URL: https://issues.apache.org/jira/browse/HBASE-5232
 Project: HBase
  Issue Type: Bug
  Components: scripts
Affects Versions: 0.94.0
Reporter: nkeywal
Priority: Minor


The HBaseBook says in chapter 2.5.1
{noformat}
If you would like to run ZooKeeper yourself, independent of HBase start/stop, 
you would do the following

${HBASE_HOME}/bin/hbase-daemons.sh {start,stop} zookeeper

Note that you can use HBase in this manner to spin up a ZooKeeper cluster, 
unrelated to HBase. Just make sure to set HBASE_MANAGES_ZK to false if you want 
it to stay up across HBase restarts so that when HBase shuts down, it doesn't 
take ZooKeeper down with it.
{noformat}

However, it does not work because the hbase-daemons script always calls 
zookeepers.sh which checks HBASE_MANAGES_ZK and does nothing when it's not set 
to true.

Imho, the best solution would be to keep the documentation and change the 
scripts.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss

[
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189339#comment-13189339
]

Hadoop QA commented on HBASE-5179:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511158/5179-90v13.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/813//console

This message is automatically generated.

Concurrent processing of processFaileOver and ServerShutdownHandler may cause
region to be assigned before log splitting is completed, causing data loss

Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch,
5179-90v12.patch, 5179-90v13.txt, 5179-90v2.patch, 5179-90v3.patch,
5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch,
5179-90v8.patch, 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt,
5179-v3.txt, 5179-v4.txt, Errorlog, hbase-5179.patch, hbase-5179v10.patch,
hbase-5179v12.patch, hbase-5179v5.patch, hbase-5179v6.patch,
hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch

[jira] [Commented] (HBASE-5231) Backport HBASE-3373 (per-table load balancing) to 0.92


[ 
https://issues.apache.org/jira/browse/HBASE-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189345#comment-13189345
 ] 

Zhihong Yu commented on HBASE-5231:
---

Some of my work in load balancer was checked into 0.92 half a year ago.
I wish customers can use the basic feature early.
HBASE-3373 being in TRUNK wouldn't mean much to majority of users.

 Backport HBASE-3373 (per-table load balancing) to 0.92
 --

 Key: HBASE-5231
 URL: https://issues.apache.org/jira/browse/HBASE-5231
 Project: HBase
  Issue Type: Improvement
Reporter: Zhihong Yu

 This JIRA backports per-table load balancing to 0.90

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5231) Backport HBASE-3373 (per-table load balancing) to 0.92

2012-01-19 Thread Jean-Daniel Cryans (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189349#comment-13189349
 ] 

Jean-Daniel Cryans commented on HBASE-5231:
---

{quote}
Some of my work in load balancer was checked into 0.92 half a year ago.
I wish customers can use the basic feature early.
{quote}

We've been testing 0.92 for about that time, this release has way too much load.

bq. HBASE-3373 being in TRUNK wouldn't mean much to majority of users.

The way it works is if you can get a +1 and no -1 then you can commit, my 
current -0 won't block you from backporting.

 Backport HBASE-3373 (per-table load balancing) to 0.92
 --

 Key: HBASE-5231
 URL: https://issues.apache.org/jira/browse/HBASE-5231
 Project: HBase
  Issue Type: Improvement
Reporter: Zhihong Yu

 This JIRA backports per-table load balancing to 0.90

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5231) Backport HBASE-3373 (per-table load balancing) to 0.92


 [ 
https://issues.apache.org/jira/browse/HBASE-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5231:
--

Attachment: 5231.txt

Patch for 0.92
The following tests pass:
{code}
  751  mt -Dtest=TestRegionRebalancing
  752  mt -Dtest=TestDefaultLoadBalancer
{code}

 Backport HBASE-3373 (per-table load balancing) to 0.92
 --

 Key: HBASE-5231
 URL: https://issues.apache.org/jira/browse/HBASE-5231
 Project: HBase
  Issue Type: Improvement
Reporter: Zhihong Yu
 Attachments: 5231.txt


 This JIRA backports per-table load balancing to 0.90

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4720) Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server


[ 
https://issues.apache.org/jira/browse/HBASE-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189400#comment-13189400
 ] 

Zhihong Yu commented on HBASE-4720:
---

@Andrew:
Can you please review patch v4 ?

 Implement atomic update operations (checkAndPut, checkAndDelete) for REST 
 client/server 
 

 Key: HBASE-4720
 URL: https://issues.apache.org/jira/browse/HBASE-4720
 Project: HBase
  Issue Type: Improvement
Reporter: Daniel Lord
Assignee: Mubarak Seyed
 Fix For: 0.94.0

 Attachments: HBASE-4720.trunk.v1.patch, HBASE-4720.trunk.v2.patch, 
 HBASE-4720.trunk.v3.patch, HBASE-4720.trunk.v4.patch, HBASE-4720.v1.patch, 
 HBASE-4720.v3.patch


 I have several large application/HBase clusters where an application node 
 will occasionally need to talk to HBase from a different cluster.  In order 
 to help ensure some of my consistency guarantees I have a sentinel table that 
 is updated atomically as users interact with the system.  This works quite 
 well for the regular hbase client but the REST client does not implement 
 the checkAndPut and checkAndDelete operations.  This exposes the application 
 to some race conditions that have to be worked around.  It would be ideal if 
 the same checkAndPut/checkAndDelete operations could be supported by the REST 
 client.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5233) [book] configuration.xml - adding example of /etc/hosts for loopback ip config

2012-01-19 Thread Doug Meil (Created) (JIRA)

[book] configuration.xml - adding example of /etc/hosts for loopback ip config
--

 Key: HBASE-5233
 URL: https://issues.apache.org/jira/browse/HBASE-5233
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor


configuration.xml - added extra paragraph in the loopback ip config section 
with an example of /etc/hosts.  The question came up again on the dist-list.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5233) [book] configuration.xml - adding example of /etc/hosts for loopback ip config


 [ 
https://issues.apache.org/jira/browse/HBASE-5233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-5233:
-

Status: Patch Available  (was: Open)

 [book] configuration.xml - adding example of /etc/hosts for loopback ip config
 --

 Key: HBASE-5233
 URL: https://issues.apache.org/jira/browse/HBASE-5233
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: configuration_hbase_5233.xml.patch


 configuration.xml - added extra paragraph in the loopback ip config section 
 with an example of /etc/hosts.  The question came up again on the dist-list.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5233) [book] configuration.xml - adding example of /etc/hosts for loopback ip config


 [ 
https://issues.apache.org/jira/browse/HBASE-5233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-5233:
-

Attachment: configuration_hbase_5233.xml.patch

 [book] configuration.xml - adding example of /etc/hosts for loopback ip config
 --

 Key: HBASE-5233
 URL: https://issues.apache.org/jira/browse/HBASE-5233
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: configuration_hbase_5233.xml.patch


 configuration.xml - added extra paragraph in the loopback ip config section 
 with an example of /etc/hosts.  The question came up again on the dist-list.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5233) [book] configuration.xml - adding example of /etc/hosts for loopback ip config


 [ 
https://issues.apache.org/jira/browse/HBASE-5233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-5233:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

 [book] configuration.xml - adding example of /etc/hosts for loopback ip config
 --

 Key: HBASE-5233
 URL: https://issues.apache.org/jira/browse/HBASE-5233
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: configuration_hbase_5233.xml.patch


 configuration.xml - added extra paragraph in the loopback ip config section 
 with an example of /etc/hosts.  The question came up again on the dist-list.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5229) Support atomic region operations

2012-01-19 Thread Lars Hofhansl (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189434#comment-13189434
]

Lars Hofhansl commented on HBASE-5229:
--

That is true when it comes to storage.
However our (current) API is mostly row based. There is no way to start or stop
a stop a scan at a column, there are many assumptions about rows baked into the
scanner, etc. I don't think ColumnRangeFilter would be good enough here.

Declaring a prefix and honoring it during splitting seems simpler and more in
line with our current API and (probably?) what a user would expect.

It is another avenue, though. For example we can add Scan.set{Start|Stop}Key
(where we can present prefixes of the full key, rather than just the row key),
and handle it accordingly at the server. Would also need a nextKeyValue (or
nextColumn or something) method on ResultScanner along with the server code
that does this efficiently.

Support atomic region operations

Attachments: 5229.txt

[jira] [Created] (HBASE-5234) [book] performance.xml - adding another paragraph on the One Hot Region entry

2012-01-19 Thread Doug Meil (Created) (JIRA)

[book] performance.xml - adding another paragraph on the One Hot Region entry
---

 Key: HBASE-5234
 URL: https://issues.apache.org/jira/browse/HBASE-5234
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: performance_hbase_5234.xml.patch

performance.xml
* adding another paragraph on the Writing One Hot Region section.  
Monotonically increasing keys is one problem already documented, but the 
paragraph I added was about pre-splitting regions and still having a hot region 
not due to increasing keys, but to bad splits.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5234) [book] performance.xml - adding another paragraph on the One Hot Region entry


 [ 
https://issues.apache.org/jira/browse/HBASE-5234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-5234:
-

Attachment: performance_hbase_5234.xml.patch

 [book] performance.xml - adding another paragraph on the One Hot Region 
 entry
 ---

 Key: HBASE-5234
 URL: https://issues.apache.org/jira/browse/HBASE-5234
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: performance_hbase_5234.xml.patch


 performance.xml
 * adding another paragraph on the Writing One Hot Region section.  
 Monotonically increasing keys is one problem already documented, but the 
 paragraph I added was about pre-splitting regions and still having a hot 
 region not due to increasing keys, but to bad splits.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5234) [book] performance.xml - adding another paragraph on the One Hot Region entry


 [ 
https://issues.apache.org/jira/browse/HBASE-5234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-5234:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

 [book] performance.xml - adding another paragraph on the One Hot Region 
 entry
 ---

 Key: HBASE-5234
 URL: https://issues.apache.org/jira/browse/HBASE-5234
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: performance_hbase_5234.xml.patch


 performance.xml
 * adding another paragraph on the Writing One Hot Region section.  
 Monotonically increasing keys is one problem already documented, but the 
 paragraph I added was about pre-splitting regions and still having a hot 
 region not due to increasing keys, but to bad splits.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5234) [book] performance.xml - adding another paragraph on the One Hot Region entry


 [ 
https://issues.apache.org/jira/browse/HBASE-5234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-5234:
-

Status: Patch Available  (was: Open)

 [book] performance.xml - adding another paragraph on the One Hot Region 
 entry
 ---

 Key: HBASE-5234
 URL: https://issues.apache.org/jira/browse/HBASE-5234
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: performance_hbase_5234.xml.patch


 performance.xml
 * adding another paragraph on the Writing One Hot Region section.  
 Monotonically increasing keys is one problem already documented, but the 
 paragraph I added was about pre-splitting regions and still having a hot 
 region not due to increasing keys, but to bad splits.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5229) Support atomic region operations

2012-01-19 Thread Philip Zeyliger (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189456#comment-13189456
 ] 

Philip Zeyliger commented on HBASE-5229:


Just to kibitz a little bit:

The approach you're proposing will only ever let you do local transactions: 
transactions clearly related to a prefix-friendly set of rows.  For example, 
if, say, inserting a tweet is transactional, but there's a global hash tag 
index, you might want to make deleting a tweet transactional in the sense that 
it'll clean up the index entry for you.  You're never going to get the index to 
be on the same region.  It's possible to implement multi-row transactions on 
top of single row transactions (by putting a write a head log there, see 
Megastore at http://research.google.com/pubs/pub36971.html).

On the other hand, if you're cool with that, and even just local transactions 
could be very useful to many users and is probably quite efficient.  As a user, 
I'd much prefer dealing with single rows, or row prefixes, than hoping that my 
rows are on the same region server.

 Support atomic region operations
 

 Key: HBASE-5229
 URL: https://issues.apache.org/jira/browse/HBASE-5229
 Project: HBase
  Issue Type: New Feature
  Components: client, regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5229.txt


 As discussed (at length) on the dev mailing list with the HBASE-3584 and 
 HBASE-5203 committed, supporting atomic cross row transactions within a 
 region becomes simple.
 I am aware of the hesitation about the usefulness of this feature, but we 
 have to start somewhere.
 Let's use this jira for discussion, I'll attach a patch (with tests) 
 momentarily to make this concrete.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5228) [REST] Rip out transform feature

2012-01-19 Thread Ben West (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189461#comment-13189461
]

Ben West commented on HBASE-5228:
-

@Andrew, Zhihong: Confirmed that I'm no longer seeing all those threads blocked
on the HTableDescriptor call.

I'm not seeing a huge performance improvement, but I switched boxes in between
tests (and am having trouble reverting to 0.90.4), so if it's faster for you
guys I'll believe that's the cause.

[REST] Rip out transform feature
--

Attachments: HBASE-5228-0.92.patch, HBASE-5228-trunk.patch

[jira] [Commented] (HBASE-5229) Support atomic region operations

2012-01-19 Thread Lars Hofhansl (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189464#comment-13189464
 ] 

Lars Hofhansl commented on HBASE-5229:
--

Thanks Philip. Yep, that's the idea with this. Global transactions would be 
handled differently and be *far* more heavyweight (and must be used judiciously 
or your load will not scale).

The transactions I have in mind with this would be just as fast as a current 
Puts/Deletes (with the drawback that related data would be collocated).


 Support atomic region operations
 

 Key: HBASE-5229
 URL: https://issues.apache.org/jira/browse/HBASE-5229
 Project: HBase
  Issue Type: New Feature
  Components: client, regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5229.txt


 As discussed (at length) on the dev mailing list with the HBASE-3584 and 
 HBASE-5203 committed, supporting atomic cross row transactions within a 
 region becomes simple.
 I am aware of the hesitation about the usefulness of this feature, but we 
 have to start somewhere.
 Let's use this jira for discussion, I'll attach a patch (with tests) 
 momentarily to make this concrete.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5229) Support atomic region operations

2012-01-19 Thread Matt Corgan (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189487#comment-13189487
]

Matt Corgan commented on HBASE-5229:

I think the approach of expanding and improving support for gigantic rows would
be cleaner than adding another consistency guarantee to hbase's feature list.
Combining very wide rows with many column families, each with separate settings
and compactions provides a good framework for all sorts of different data
models on top. You could let a row become so big that it's the only row on a
machine if you want.

From what I can tell from papers and presentations, BigTable actually supports
many gigabyte EntityGroups. They mention it at 6:18 in this video:
http://www.youtube.com/watch?v=xO015C3R6dw . I'm not sure how Entity Groups
can span machines and still enforce transactionality. I had thought that an
entity group was confined to a single BigTable row, maybe that means they do
span BigTable regions. Anyone know how that works?

Support atomic region operations

Attachments: 5229.txt

[jira] [Resolved] (HBASE-3976) Disable Block Cache On Compactions

2012-01-19 Thread Nicolas Spiegelberg (Resolved) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Nicolas Spiegelberg resolved HBASE-3976.

Resolution: Incomplete

its very hard to tell if this issue is corrected in trunk or not. Talked
offline with Mikhail about strategies for getting proper resolution here.
HBASE-5230 has been made to verify that this issue is indeed fixed will
encompass a fix if other subtle bugs remain.

Disable Block Cache On Compactions
--

Key: HBASE-3976
URL: https://issues.apache.org/jira/browse/HBASE-3976
Project: HBase
Issue Type: Improvement
Components: regionserver
Affects Versions: 0.90.3
Reporter: Karthick Sankarachary
Assignee: Nicolas Spiegelberg
Priority: Minor
Attachments: HBASE-3976-V3.patch, HBASE-3976-unconditional.patch,
HBASE-3976.patch

Is there a good reason to believe that caching blocks during compactions is
beneficial? Currently, if block cache is enabled on a certain family, then
every time it's compacted, we load all of its blocks into the (LRU) cache, at
the expense of the legitimately hot ones.
As a matter of fact, this concern was raised earlier in HBASE-1597, which
rightly points out that, we should not bog down the LRU with unneccessary
blocks during compaction. Even though that issue has been marked as fixed,
it looks like it ought to be reopened.
Should we err on the side of caution and not cache blocks during compactions
period (as illustrated in the attached patch)? Or, can we be selectively
aggressive about what blocks do get cached during compaction (e.g., only
cache those blocks from the recent files)?

[jira] [Commented] (HBASE-5233) [book] configuration.xml - adding example of /etc/hosts for loopback ip config


[ 
https://issues.apache.org/jira/browse/HBASE-5233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189506#comment-13189506
 ] 

Hudson commented on HBASE-5233:
---

Integrated in HBase-TRUNK #2640 (See 
[https://builds.apache.org/job/HBase-TRUNK/2640/])
hbase-5233. [book] configuration.xml - adding example of /etc/hosts for 
loopback ip config


 [book] configuration.xml - adding example of /etc/hosts for loopback ip config
 --

 Key: HBASE-5233
 URL: https://issues.apache.org/jira/browse/HBASE-5233
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: configuration_hbase_5233.xml.patch


 configuration.xml - added extra paragraph in the loopback ip config section 
 with an example of /etc/hosts.  The question came up again on the dist-list.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5231) Backport HBASE-3373 (per-table load balancing) to 0.92


[ 
https://issues.apache.org/jira/browse/HBASE-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189507#comment-13189507
 ] 

Zhihong Yu commented on HBASE-5231:
---

Ran the patch over 0.92 test suite.

All tests passed.

 Backport HBASE-3373 (per-table load balancing) to 0.92
 --

 Key: HBASE-5231
 URL: https://issues.apache.org/jira/browse/HBASE-5231
 Project: HBase
  Issue Type: Improvement
Reporter: Zhihong Yu
 Attachments: 5231.txt


 This JIRA backports per-table load balancing to 0.90

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4608) HLog Compression


[ 
https://issues.apache.org/jira/browse/HBASE-4608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189520#comment-13189520
 ] 

Zhihong Yu commented on HBASE-4608:
---

I tried to run TestWALReplayCompressed:
{code}
Running org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.008 sec

Results :

Tests run: 5, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- maven-surefire-plugin:2.10:test (secondPartTestsExecution) @ hbase 
---
[INFO] Tests are skipped.
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 3:50.838s
{code}
Looks like the ShutdownHooks took a long time to finish:
{code}
main prio=5 tid=104000800 nid=0x100601000 in Object.wait() [10060]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1210)
- locked 78e887470 (a org.apache.hadoop.fs.FileSystem$ClientFinalizer)
at java.lang.Thread.join(Thread.java:1263)
at 
java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
at 
java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
at java.lang.Shutdown.runHooks(Shutdown.java:79)
at java.lang.Shutdown.sequence(Shutdown.java:123)
at java.lang.Shutdown.exit(Shutdown.java:168)
- locked 7faf9d288 (a java.lang.Class for java.lang.Shutdown)
at java.lang.Runtime.exit(Runtime.java:90)
at java.lang.System.exit(System.java:921)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:73)
{code}

 HLog Compression
 

 Key: HBASE-4608
 URL: https://issues.apache.org/jira/browse/HBASE-4608
 Project: HBase
  Issue Type: New Feature
Reporter: Li Pi
Assignee: Li Pi
 Attachments: 4608v1.txt, 4608v5.txt, 4608v6.txt, 4608v7.txt, 
 4608v8fixed.txt


 The current bottleneck to HBase write speed is replicating the WAL appends 
 across different datanodes. We can speed up this process by compressing the 
 HLog. Current plan involves using a dictionary to compress table name, region 
 id, cf name, and possibly other bits of repeated data. Also, HLog format may 
 be changed in other ways to produce a smaller HLog.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4608) HLog Compression

2012-01-19 Thread Li Pi (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189530#comment-13189530
 ] 

Li Pi commented on HBASE-4608:
--

Are the shutdown hooks slower than TestWALReplay without compression?

On Thu, Jan 19, 2012 at 4:12 PM, Zhihong Yu (Commented) (JIRA)


 HLog Compression
 

 Key: HBASE-4608
 URL: https://issues.apache.org/jira/browse/HBASE-4608
 Project: HBase
  Issue Type: New Feature
Reporter: Li Pi
Assignee: Li Pi
 Attachments: 4608v1.txt, 4608v5.txt, 4608v6.txt, 4608v7.txt, 
 4608v8fixed.txt


 The current bottleneck to HBase write speed is replicating the WAL appends 
 across different datanodes. We can speed up this process by compressing the 
 HLog. Current plan involves using a dictionary to compress table name, region 
 id, cf name, and possibly other bits of repeated data. Also, HLog format may 
 be changed in other ways to produce a smaller HLog.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4608) HLog Compression


[ 
https://issues.apache.org/jira/browse/HBASE-4608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189534#comment-13189534
 ] 

Zhihong Yu commented on HBASE-4608:
---

Similar result:
{code}
Running org.apache.hadoop.hbase.regionserver.wal.TestWALReplay
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.865 sec

Results :

Tests run: 5, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- maven-surefire-plugin:2.10:test (secondPartTestsExecution) @ hbase 
---
[INFO] Tests are skipped.
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 3:46.105s
{code}

 HLog Compression
 

 Key: HBASE-4608
 URL: https://issues.apache.org/jira/browse/HBASE-4608
 Project: HBase
  Issue Type: New Feature
Reporter: Li Pi
Assignee: Li Pi
 Attachments: 4608v1.txt, 4608v5.txt, 4608v6.txt, 4608v7.txt, 
 4608v8fixed.txt


 The current bottleneck to HBase write speed is replicating the WAL appends 
 across different datanodes. We can speed up this process by compressing the 
 HLog. Current plan involves using a dictionary to compress table name, region 
 id, cf name, and possibly other bits of repeated data. Also, HLog format may 
 be changed in other ways to produce a smaller HLog.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5228) [REST] Rip out transform feature

2012-01-19 Thread Andrew Purtell (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189556#comment-13189556
]

Andrew Purtell commented on HBASE-5228:
---

There could be more going on, but let's get this problem settled and keep
digging. I'll commit to trunk, 0.92, and 0.90 tonight after the meetup.

[REST] Rip out transform feature
--

Attachments: HBASE-5228-0.92.patch, HBASE-5228-trunk.patch

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss

2012-01-19 Thread chunhui shen (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189578#comment-13189578
 ] 

chunhui shen commented on HBASE-5179:
-

@Jinchao 
{code}
gaojinchao added a comment - 19/Jan/12 16:40 
when region server checkin, I killed meta/root region. I found splitHlog is 
after meta was assigned. So I think may has another problem. tomorrow continue 
to dig .
{code}

Because this test uses v11patch without waitUntilNoLogDir(), so it is the 
problem .

Jinchao, could you test again with 90v13.patch.
Thanks.

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v13.txt, 5179-90v2.patch, 5179-90v3.patch, 
 5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 
 5179-90v8.patch, 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 
 5179-v3.txt, 5179-v4.txt, Errorlog, hbase-5179.patch, hbase-5179v10.patch, 
 hbase-5179v12.patch, hbase-5179v5.patch, hbase-5179v6.patch, 
 hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss


 [ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chunhui shen updated HBASE-5179:


Attachment: 5179-90v14.patch

90v13 has a mistake about waitUntilNoLogDir
{code}if (this.fileSystemManager.logDirExists(serverName) != null) {
break;
  }{code}
should changed as 
{code}
if (this.fileSystemManager.logDirExists(serverName) == null) {
break;
  }{code}


also prevent from NPE about it.

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v13.txt, 5179-90v14.patch, 5179-90v2.patch, 
 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 
 5179-90v7.patch, 5179-90v8.patch, 5179-90v9.patch, 5179-v11-92.txt, 
 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, Errorlog, 
 hbase-5179.patch, hbase-5179v10.patch, hbase-5179v12.patch, 
 hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, 
 hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss

[
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189585#comment-13189585
]

Hadoop QA commented on HBASE-5179:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511216/5179-90v14.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/814//console

This message is automatically generated.

Concurrent processing of processFaileOver and ServerShutdownHandler may cause
region to be assigned before log splitting is completed, causing data loss

Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch,
5179-90v12.patch, 5179-90v13.txt, 5179-90v14.patch, 5179-90v2.patch,
5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch,
5179-90v7.patch, 5179-90v8.patch, 5179-90v9.patch, 5179-v11-92.txt,
5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, Errorlog,
hbase-5179.patch, hbase-5179v10.patch, hbase-5179v12.patch,
hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch,
hbase-5179v8.patch, hbase-5179v9.patch

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss

2012-01-19 Thread chunhui shen (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189584#comment-13189584
 ] 

chunhui shen commented on HBASE-5179:
-

@Jinchao 
Could you test again with 90v14patch?
Thanks much!

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v13.txt, 5179-90v14.patch, 5179-90v2.patch, 
 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 
 5179-90v7.patch, 5179-90v8.patch, 5179-90v9.patch, 5179-v11-92.txt, 
 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, Errorlog, 
 hbase-5179.patch, hbase-5179v10.patch, hbase-5179v12.patch, 
 hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, 
 hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss


[ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189595#comment-13189595
 ] 

gaojinchao commented on HBASE-5179:
---

In my test case, I kill meta/root at same time. when master start to assign 
root/meta region , it should finish split hlogs. 

Today I will give a detail test report about 90V14.


 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v13.txt, 5179-90v14.patch, 5179-90v2.patch, 
 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 
 5179-90v7.patch, 5179-90v8.patch, 5179-90v9.patch, 5179-v11-92.txt, 
 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, Errorlog, 
 hbase-5179.patch, hbase-5179v10.patch, hbase-5179v12.patch, 
 hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, 
 hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss


 [ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chunhui shen updated HBASE-5179:


Attachment: 5179-90v15.patch

@Jinchao
I find 90V14 has also a problem.
I'm sorry.
Could you use v15.
Thanks. 

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v13.txt, 5179-90v14.patch, 5179-90v15.patch, 
 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 
 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 5179-90v9.patch, 
 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, 
 Errorlog, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v12.patch, 
 hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, 
 hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss


[ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189600#comment-13189600
 ] 

gaojinchao commented on HBASE-5179:
---

no problem. before I start to verify the patch. I will review it.

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v13.txt, 5179-90v14.patch, 5179-90v15.patch, 
 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 
 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 5179-90v9.patch, 
 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, 
 Errorlog, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v12.patch, 
 hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, 
 hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss


[ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189607#comment-13189607
 ] 

Hadoop QA commented on HBASE-5179:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12511218/5179-90v15.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

-1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/815//console

This message is automatically generated.

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v13.txt, 5179-90v14.patch, 5179-90v15.patch, 
 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 
 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 5179-90v9.patch, 
 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, 
 Errorlog, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v12.patch, 
 hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, 
 hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4218) Data Block Encoding of KeyValues (aka delta encoding / prefix compression)

2012-01-19 Thread Phabricator (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189623#comment-13189623
]

Phabricator commented on HBASE-4218:

Kannan has commented on the revision [jira] [HBASE-4218] HFile data block
encoding framework and delta encoding implementation.

a few more minor comments. I still have a few more files left to review.

INLINE COMMENTS

src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java:45
rename: offset - prevOffset
src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java:158 we can
avoid this duplication, and instead call the other load of copyToStream, and
pass offset = 0.
src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java:192 length
name is confusing for 2nd argument. This function can put any long. Rename
length - value; and tmpLength - tmpValue.
src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java:471 There are
4 overloads of the readCompressedInt().

The first 2 work on a 7-bit encoding scheme.

The next 2 work on a different format of encoding. These overloads are also
unused other than tests. Let's remove these two overloads.

REVISION DETAIL
https://reviews.facebook.net/D447

Data Block Encoding of KeyValues (aka delta encoding / prefix compression)
---

Key: HBASE-4218
URL: https://issues.apache.org/jira/browse/HBASE-4218
Project: HBase
Issue Type: Improvement
Components: io
Affects Versions: 0.94.0
Reporter: Jacek Migdal
Assignee: Mikhail Bautin
Labels: compression
Fix For: 0.94.0

Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch,
0001-Delta-encoding.patch, 4218-2012-01-14.txt, 4218-v16.txt, 4218.txt,
D447.1.patch, D447.10.patch, D447.11.patch, D447.12.patch, D447.13.patch,
D447.14.patch, D447.15.patch, D447.16.patch, D447.17.patch, D447.18.patch,
D447.19.patch, D447.2.patch, D447.20.patch, D447.21.patch, D447.22.patch,
D447.23.patch, D447.24.patch, D447.3.patch, D447.4.patch, D447.5.patch,
D447.6.patch, D447.7.patch, D447.8.patch, D447.9.patch,
Data-block-encoding-2011-12-23.patch,
Delta-encoding-2012-01-17_11_09_09.patch,
Delta-encoding.patch-2011-12-22_11_52_07.patch,
Delta-encoding.patch-2012-01-05_15_16_43.patch,
Delta-encoding.patch-2012-01-05_16_31_44.patch,
Delta-encoding.patch-2012-01-05_16_31_44_copy.patch,
Delta-encoding.patch-2012-01-05_18_50_47.patch,
Delta-encoding.patch-2012-01-07_14_12_48.patch,
Delta-encoding.patch-2012-01-13_12_20_07.patch,
Delta_encoding_with_memstore_TS.patch, open-source.diff

A compression for keys. Keys are sorted in HFile and they are usually very
similar. Because of that, it is possible to design better compression than
general purpose algorithms,
It is an additional step designed to be used in memory. It aims to save
memory in cache as well as speeding seeks within HFileBlocks. It should
improve performance a lot, if key lengths are larger than value lengths. For
example, it makes a lot of sense to use it when value is a counter.
Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes)
shows that I could achieve decent level of compression:
key compression ratio: 92%
total compression ratio: 85%
LZO on the same data: 85%
LZO after delta encoding: 91%
While having much better performance (20-80% faster decompression ratio than
LZO). Moreover, it should allow far more efficient seeking which should
improve performance a bit.
It seems that a simple compression algorithms are good enough. Most of the
savings are due to prefix compression, int128 encoding, timestamp diffs and
bitfields to avoid duplication. That way, comparisons of compressed data can
be much faster than a byte comparator (thanks to prefix compression and
bitfields).
In order to implement it in HBase two important changes in design will be
needed:
-solidify interface to HFileBlock / HFileReader Scanner to provide seeking
and iterating; access to uncompressed buffer in HFileBlock will have bad
performance
-extend comparators to support comparison assuming that N first bytes are
equal (or some fields are equal)
Link to a discussion about something similar:
http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windowssubj=Re+prefix+compression

[jira] [Updated] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss

[
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Zhihong Yu updated HBASE-5179:
--

Comment: was deleted

(was: -1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511216/5179-90v14.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/814//console

This message is automatically generated.)

Concurrent processing of processFaileOver and ServerShutdownHandler may cause
region to be assigned before log splitting is completed, causing data loss

Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch,
5179-90v12.patch, 5179-90v13.txt, 5179-90v14.patch, 5179-90v15.patch,
5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch,
5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 5179-90v9.patch,
5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt,
Errorlog, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v12.patch,
hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch,
hbase-5179v8.patch, hbase-5179v9.patch

[jira] [Updated] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss

[
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Zhihong Yu updated HBASE-5179:
--

Comment: was deleted

(was: -1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511218/5179-90v15.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/815//console

This message is automatically generated.)

Concurrent processing of processFaileOver and ServerShutdownHandler may cause
region to be assigned before log splitting is completed, causing data loss

[jira] [Commented] (HBASE-5227) [book] added warning about using RowLocks


[ 
https://issues.apache.org/jira/browse/HBASE-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189638#comment-13189638
 ] 

Hudson commented on HBASE-5227:
---

Integrated in HBase-TRUNK-security #82 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/82/])
hbase-5227.  [book] book.xml, troubleshooting.xml - added caution about 
rowlocks.


 [book] added warning about using RowLocks
 -

 Key: HBASE-5227
 URL: https://issues.apache.org/jira/browse/HBASE-5227
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: docbkx_hbase_5227.patch


 book.xml
 * Arch/Client - added entry discouraging users from RowLock, with link to 
 outstanding Jira on removing the feature entirely.
 troubleshooting.xml 
 * RS Runtime section.  Under existing RS Hanging entry added a paragraph 
 pointing to the newly creatd rowlock entry in the Arch/Client section.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5233) [book] configuration.xml - adding example of /etc/hosts for loopback ip config


[ 
https://issues.apache.org/jira/browse/HBASE-5233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189636#comment-13189636
 ] 

Hudson commented on HBASE-5233:
---

Integrated in HBase-TRUNK-security #82 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/82/])
hbase-5233. [book] configuration.xml - adding example of /etc/hosts for 
loopback ip config


 [book] configuration.xml - adding example of /etc/hosts for loopback ip config
 --

 Key: HBASE-5233
 URL: https://issues.apache.org/jira/browse/HBASE-5233
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: configuration_hbase_5233.xml.patch


 configuration.xml - added extra paragraph in the loopback ip config section 
 with an example of /etc/hosts.  The question came up again on the dist-list.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5208) Allow setting Scan start/stop row individually in TableInputFormat


[ 
https://issues.apache.org/jira/browse/HBASE-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189639#comment-13189639
 ] 

Hudson commented on HBASE-5208:
---

Integrated in HBase-TRUNK-security #82 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/82/])
HBASE-5208 Allow setting Scan start/stop row individually in 
TableInputFormat

tedyu : 
Files : 
* /hbase/trunk/CHANGES.txt
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormat.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScan.java


 Allow setting Scan start/stop row individually in TableInputFormat
 --

 Key: HBASE-5208
 URL: https://issues.apache.org/jira/browse/HBASE-5208
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Reporter: Nicholas Telford
Priority: Minor
 Attachments: HBASE-5208-001.txt, HBASE-5208-002.txt, 
 HBASE-5208-003.txt, HBASE-5208-004.txt


 Currently, TableInputFormat initializes a serialized Scan from 
 hbase.mapreduce.scan. Alternatively, it will instantiate a new Scan using 
 properties defined in hbase.mapreduce.scan.*. However, of these properties 
 the start row and stop row (arguably the most pertinent) are missing.
 TableInputFormat should permit the specification of a start/stop row as with 
 the other fields using a new pair of properties: 
 hbase.mapreduce.scan.row.start and hbase.mapreduce.scan.row.end
 The primary use-case for this is to permit Oozie and other job management 
 tools that can't call TableMapReduceUtil.initTableMapperJob() to operate on a 
 contiguous subset of rows.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5226) [book] troubleshooting.xml - add client slowdown when calling admin APIs issue to Client section


[ 
https://issues.apache.org/jira/browse/HBASE-5226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189637#comment-13189637
 ] 

Hudson commented on HBASE-5226:
---

Integrated in HBase-TRUNK-security #82 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/82/])
hbase-5226.  troubleshooting.xml, client section:  adding reference to 
client slowdown due to admin API calls.


 [book] troubleshooting.xml - add client slowdown when calling admin APIs 
 issue to Client section
 

 Key: HBASE-5226
 URL: https://issues.apache.org/jira/browse/HBASE-5226
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: troubleshooting_hbase_5226.xml.patch


 troubleshooting.xml - adding client slowdown when calling admin APIs issue 
 (HBASE-5073) to the Client section.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5223) TestMetaReaderEditor is missing call to CatalogTracker.stop()


[ 
https://issues.apache.org/jira/browse/HBASE-5223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189635#comment-13189635
 ] 

Hudson commented on HBASE-5223:
---

Integrated in HBase-TRUNK-security #82 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/82/])
HBASE-5223  TestMetaReaderEditor is missing call to CatalogTracker.stop()

tedyu : 
Files : 
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java


 TestMetaReaderEditor is missing call to CatalogTracker.stop()
 -

 Key: HBASE-5223
 URL: https://issues.apache.org/jira/browse/HBASE-5223
 Project: HBase
  Issue Type: Test
Reporter: Zhihong Yu
 Fix For: 0.94.0, 0.92.1

 Attachments: 5223.txt


 I noticed that TestMetaReaderEditor hung on 0.92 Jenkins builds - see 
 https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/249/console
 It turns out that CatalogTracker.stop() is missing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss


[ 
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189641#comment-13189641
 ] 

gaojinchao commented on HBASE-5179:
---

@chunhui
The first test case failed, we start a cluster, the patch is split a new region 
server's Hlog.

2012-01-20 00:34:39,462 INFO org.mortbay.log: Started 
SelectChannelConnector@0.0.0.0:20010
2012-01-20 00:34:39,462 DEBUG org.apache.hadoop.hbase.master.HMaster: Started 
service threads
2012-01-20 00:34:40,158 INFO org.apache.hadoop.hbase.master.ServerManager: 
Registering server=C3S32,20020,1327037679721, regionCount=0, userLoad=false
2012-01-20 00:34:40,296 INFO org.apache.hadoop.hbase.master.ServerManager: 
Registering server=C3S33,20020,1327037679059, regionCount=0, userLoad=false
2012-01-20 00:34:40,488 INFO org.apache.hadoop.hbase.master.ServerManager: 
Registering server=C3S31,20020,1327037679673, regionCount=0, userLoad=false
2012-01-20 00:34:40,962 INFO org.apache.hadoop.hbase.master.ServerManager: 
Waiting on regionserver(s) count to settle; currently=3
2012-01-20 00:34:42,462 INFO org.apache.hadoop.hbase.master.ServerManager: 
Finished waiting for regionserver count to settle; count=3, sleptFor=3000
2012-01-20 00:34:42,463 INFO org.apache.hadoop.hbase.master.ServerManager: 
Exiting wait on regionserver(s) to checkin; count=3, stopped=false, count of 
regions out on cluster=0
2012-01-20 00:34:42,463 INFO org.apache.hadoop.hbase.master.HMaster: 
--sleep 60s-
2012-01-20 00:35:42,469 INFO org.apache.hadoop.hbase.master.MasterFileSystem: 
Log folder hdfs://C3S31:9000/hbase/.logs/C3S31,20020,1327037679673 belongs to 
an existing region server
2012-01-20 00:35:42,470 INFO org.apache.hadoop.hbase.master.MasterFileSystem: 
Log folder hdfs://C3S31:9000/hbase/.logs/C3S32,20020,1327037679721 belongs to 
an existing region server
2012-01-20 00:35:42,470 INFO org.apache.hadoop.hbase.master.MasterFileSystem: 
Log folder hdfs://C3S31:9000/hbase/.logs/C3S33,20020,1327037679059 belongs to 
an existing region server
2012-01-20 00:35:42,504 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: 
Failed verification of -ROOT-,,0 at address=C3S32:20020; 
org.apache.hadoop.hbase.NotServingRegionException: 
org.apache.hadoop.hbase.NotServingRegionException: Region is not online: 
-ROOT-,,0
2012-01-20 00:36:42,610 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled 
exception. Starting shutdown.
java.lang.RuntimeException: Timed out waiting to finish splitting log for 
C3S32,20020,1327037679721
at 
org.apache.hadoop.hbase.master.HMaster.waitUntilNoLogDir(HMaster.java:578)
at 
org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:478)
at 
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:422)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:283)
2012-01-20 00:36:42,613 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2012-01-20 00:36:42,613 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping 
service threads
2012-01-20 00:36:42,613 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server 
on 2

 Concurrent processing of processFaileOver and ServerShutdownHandler may cause 
 region to be assigned before log splitting is completed, causing data loss
 

 Key: HBASE-5179
 URL: https://issues.apache.org/jira/browse/HBASE-5179
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.92.0, 0.94.0, 0.90.6

 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 
 5179-90v12.patch, 5179-90v13.txt, 5179-90v14.patch, 5179-90v15.patch, 
 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 
 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 5179-90v9.patch, 
 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, 
 Errorlog, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v12.patch, 
 hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, 
 hbase-5179v8.patch, hbase-5179v9.patch


 If master's processing its failover and ServerShutdownHandler's processing 
 happen concurrently, it may appear following  case.
 1.master completed splitLogAfterStartup()
 2.RegionserverA restarts, and ServerShutdownHandler is processing.
 3.master starts to rebuildUserRegions, and RegionserverA is considered as 
 dead server.
 4.master starts to assign regions of RegionserverA because it is a dead 
 server by step3.
 However, when doing step4(assigning region), ServerShutdownHandler may be 
 doing split log, Therefore, it may cause data loss.

--
This

[jira] [Updated] (HBASE-5223) TestMetaReaderEditor is missing call to CatalogTracker.stop()


 [ 
https://issues.apache.org/jira/browse/HBASE-5223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5223:
--

Resolution: Fixed
  Assignee: Zhihong Yu
Status: Resolved  (was: Patch Available)

 TestMetaReaderEditor is missing call to CatalogTracker.stop()
 -

 Key: HBASE-5223
 URL: https://issues.apache.org/jira/browse/HBASE-5223
 Project: HBase
  Issue Type: Test
Reporter: Zhihong Yu
Assignee: Zhihong Yu
 Fix For: 0.94.0, 0.92.1

 Attachments: 5223.txt


 I noticed that TestMetaReaderEditor hung on 0.92 Jenkins builds - see 
 https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/249/console
 It turns out that CatalogTracker.stop() is missing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5231) Backport HBASE-3373 (per-table load balancing) to 0.92