date:20140604


[ 
https://issues.apache.org/jira/browse/HBASE-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017410#comment-14017410
 ] 

stack commented on HBASE-10641:
---

I took a quick look.  +1

 Configurable Bucket Sizes in bucketCache
 

 Key: HBASE-10641
 URL: https://issues.apache.org/jira/browse/HBASE-10641
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Biju Nair
Assignee: Nick Dimiduk
  Labels: cache
 Fix For: 0.99.0, 0.98.4

 Attachments: HBASE-10641.00.patch, HBASE-10641.01-0.98.patch, 
 HBASE-10641.01.patch, HBASE-10641.02-0.98.patch, HBASE-10641.02.patch


 In the current implementation of BucketCache, the sizes of buckets created 
 for different blocksizes is fixed. Given that the all the blocksizes will not 
 be used, it would be good to have the bucketCache configurable. The block 
 sizes for which buckets need to be created and the size of the buckets for 
 each block size should be configurable.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11280) Document distributed log replay and distributed log splitting


 [ 
https://issues.apache.org/jira/browse/HBASE-11280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misty Stanley-Jones updated HBASE-11280:


Hadoop Flags:   (was: Reviewed)
  Status: Patch Available  (was: Open)

 Document distributed log replay and distributed log splitting
 -

 Key: HBASE-11280
 URL: https://issues.apache.org/jira/browse/HBASE-11280
 Project: HBase
  Issue Type: Sub-task
  Components: documentation, MTTR, wal
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
 Fix For: 0.99.0

 Attachments: HBASE-11280.patch


 Enable 'distributed log replay' by default.  Depends on hfilev3 being enabled.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11280) Document distributed log replay and distributed log splitting


 [ 
https://issues.apache.org/jira/browse/HBASE-11280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misty Stanley-Jones updated HBASE-11280:


Attachment: HBASE-11280.patch

First pass at documenting distributed log splitting and distributed log replay. 
Feedback is appreciated, including feedback on whether the  chosen location is 
appropriate.

 Document distributed log replay and distributed log splitting
 -

 Key: HBASE-11280
 URL: https://issues.apache.org/jira/browse/HBASE-11280
 Project: HBase
  Issue Type: Sub-task
  Components: documentation, MTTR, wal
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
 Fix For: 0.99.0

 Attachments: HBASE-11280.patch


 Enable 'distributed log replay' by default.  Depends on hfilev3 being enabled.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11280) Document distributed log replay and distributed log splitting


[ 
https://issues.apache.org/jira/browse/HBASE-11280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017517#comment-14017517
 ] 

Hadoop QA commented on HBASE-11280:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12648295/HBASE-11280.patch
  against trunk revision .
  ATTACHMENT ID: 12648295

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+0 tests included{color}.  The patch appears to be a 
documentation patch that doesn't require tests.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+  
xlink:href=http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/regionserver/wal/HLog.html;HLog/link.
+  paraThe WAL resides in HDFS in the 
filename/hbase/.logs//filename directory, with subdirectories per
+  paraLog splitting is done by the HMaster during cluster start-up 
or by the ServerShutdownHandler
+file. The temporary edit file is stored to disk with the 
following naming pattern:/para
+  para When the region is opened, the 
filenamerecovered.edits/filename folder is checked for recovered
+(link 
xlink:href=https://issues.apache.org/jira/browse/HBASE-1364;HBASE-1364/link)
 
+[hdfs%3A%2F%2Fhost2.sample.com%3A56020%2Fhbase%2F.logs%2Fhost8.sample.com%2C57020%2C1340474893275-splitting%2Fhost8.sample.com%253A57020.1340474893900,
 
+hdfs%3A%2F%2Fhost2.sample.com%3A56020%2Fhbase%2F.logs%2Fhost3.sample.com%2C57020%2C1340474893299-splitting%2Fhost3.sample.com%253A57020.1340474893931,
 
+hdfs%3A%2F%2Fhost2.sample.com%3A56020%2Fhbase%2F.logs%2Fhost4.sample.com%2C57020%2C1340474893287-splitting%2Fhost4.sample.com%253A57020.1340474893946]
  
+userinputget 
/hbase/splitlog/hdfs%3A%2F%2Fhost2.sample.com%3A56020%2Fhbase%2F.logs%2Fhost6.sample.com%2C57020%2C1340474893287-splitting%2Fhost6.sample.com%253A57020.1340474893945

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   org.apache.hadoop.hbase.client.TestMultiParallel
  org.apache.hadoop.hbase.regionserver.TestHRegion

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9688//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9688//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9688//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9688//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9688//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9688//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9688//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9688//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9688//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9688//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9688//console

This message is automatically generated.

 Document distributed log replay and distributed log splitting
 -

 Key: HBASE-11280
 URL: https://issues.apache.org/jira/browse/HBASE-11280
 Project: HBase
  Issue Type: Sub-task
  Components: documentation, MTTR, wal
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
 Fix For: 0.99.0

 Attachments: HBASE-11280.patch


 Enable 'distributed log replay' by default.  Depends on hfilev3 being enabled.



--
This

[jira] [Commented] (HBASE-7912) HBase Backup/Restore Based on HBase Snapshot

2014-06-04 Thread Honghua Feng (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017538#comment-14017538
]

Honghua Feng commented on HBASE-7912:
-

Just finished reading the design doc
HBaseBackupRestore-Jira-7912-DesignDoc-v1.pdf. It's a good enhancement and
extension to current data backup/restore option/solution, and the design doc
reads quite concise and clear :-)

Some comments:
# Use case example 1 in page 3: The full backup doesn't contain data of
table3 and table4, so when restoring table3 and table4, their data are all
restored from the incremental backups, right? Sounds it's not a typical
scenario(full-backup + incremental backups) for backup/restore.
# 4. Full Backup: Does log roll take place after taking (full) snapshot? What
if new writes arrive after taking snapshot but before log roll?
# 5. Incremental Backup: What if some RS fails during the log roll procedure
so that not all current log number are recorded onto ZooKeeper?
# What if some log files are archived/deleted between two incremental backups
and are not included in any incremental backup? Is it possible?

Some (possible) typos in the design doc:
# 2. Key features and Use Cases: Full back uses HBase... = Full backup
uses HBase...
# 5. Incremental Backup: kicks of a global... = kicks off a global...
# 5. Incremental Backup: Incremental backups and also be... = Incremental
backups can also be...

HBase Backup/Restore Based on HBase Snapshot

Key: HBASE-7912
URL: https://issues.apache.org/jira/browse/HBASE-7912
Project: HBase
Issue Type: Sub-task
Reporter: Richard Ding
Assignee: Richard Ding
Attachments: HBaseBackupRestore-Jira-7912-DesignDoc-v1.pdf,
HBase_BackupRestore-Jira-7912-CLI-v1.pdf

Finally, we completed the implementation of our backup/restore solution, and
would like to share with community through this jira.
We are leveraging existing hbase snapshot feature, and provide a general
solution to common users. Our full backup is using snapshot to capture
metadata locally and using exportsnapshot to move data to another cluster;
the incremental backup is using offline-WALplayer to backup HLogs; we also
leverage global distribution rolllog and flush to improve performance; other
added-on values such as convert, merge, progress report, and CLI commands. So
that a common user can backup hbase data without in-depth knowledge of hbase.
Our solution also contains some usability features for enterprise users.
The detail design document and CLI command will be attached in this jira. We
plan to use 10~12 subtasks to share each of the following features, and
document the detail implement in the subtasks:
* *Full Backup* : provide local and remote back/restore for a list of tables
* *offline-WALPlayer* to convert HLog to HFiles offline (for incremental
backup)
* *distributed* Logroll and distributed flush
* Backup *Manifest* and history
* *Incremental* backup: to build on top of full backup as daily/weekly backup
* *Convert* incremental backup WAL files into hfiles
* *Merge* several backup images into one(like merge weekly into monthly)
* *add and remove* table to and from Backup image
* *Cancel* a backup process
* backup progress *status*
* full backup based on *existing snapshot*
*-*
*Below is the original description, to keep here as the history for the
design and discussion back in 2013*
There have been attempts in the past to come up with a viable HBase
backup/restore solution (e.g., HBASE-4618). Recently, there are many
advancements and new features in HBase, for example, FileLink, Snapshot, and
Distributed Barrier Procedure. This is a proposal for a backup/restore
solution that utilizes these new features to achieve better performance and
consistency.

[jira] [Updated] (HBASE-11204) Document bandwidth consumption limit feature for ExportSnapshot

2014-06-04 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-11204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-11204:
---

Resolution: Fixed
Status: Resolved  (was: Patch Available)

 Document bandwidth consumption limit feature for ExportSnapshot
 ---

 Key: HBASE-11204
 URL: https://issues.apache.org/jira/browse/HBASE-11204
 Project: HBase
  Issue Type: Task
  Components: snapshots
Reporter: Ted Yu
Assignee: Misty Stanley-Jones
Priority: Minor
 Fix For: 0.99.0

 Attachments: HBASE-11204-1.patch, HBASE-11204-2.patch, 
 HBASE-11204.patch


 http://hbase.apache.org/book.html#ops.snapshots.export should document 
 bandwidth consumption limit feature which is implemented by HBASE-11083 and 
 HBASE-11090



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10871) Indefinite OPEN/CLOSE wait on busy RegionServers

2014-06-04 Thread Jimmy Xiang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-10871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017722#comment-14017722
 ] 

Jimmy Xiang commented on HBASE-10871:
-

Yes, we should let it retry: reduce the retry counter by 1, and continue, till 
it's not a SocketTimeoutException any more. We can't count on timeout monitor, 
can't assume the server is dead.

 Indefinite OPEN/CLOSE wait on busy RegionServers
 

 Key: HBASE-10871
 URL: https://issues.apache.org/jira/browse/HBASE-10871
 Project: HBase
  Issue Type: Improvement
  Components: Balancer, master, Region Assignment
Affects Versions: 0.94.6
Reporter: Harsh J

 We observed a case where, when a specific RS got bombarded by a large amount 
 of regular requests, spiking and filling up its RPC queue, the balancer's 
 invoked unassigns and assigns for regions that dealt with this server entered 
 into an indefinite retry loop.
 The regions specifically began waiting in PENDING_CLOSE/PENDING_OPEN states 
 indefinitely cause of the HBase Client RPC from the ServerManager at the 
 master was running into SocketTimeouts. This caused a region unavailability 
 in the server for the affected regions. The timeout monitor retry default of 
 30m in 0.94's AM compounded the waiting gap further a bit more (this is now 
 10m in 0.95+'s new AM, and has further retries before we get there, which is 
 good).
 Wonder if there's a way to improve this situation generally. PENDING_OPENs 
 may be easy to handle - we can switch them out and move them elsewhere. 
 PENDING_CLOSEs may be a bit more tricky, but there must perhaps at least be a 
 way to give up permanently on a movement plan, and letting things be for a 
 while hoping for the RS to recover itself on its own (such that clients also 
 have a chance of getting things to work in the meantime)?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11295) Long running scan produces OutOfOrderScannerNextException

2014-06-04 Thread Jeff Cunningham (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017749#comment-14017749
 ] 

Jeff Cunningham commented on HBASE-11295:
-

This may be related to implementation of 
https://issues.apache.org/jira/browse/HBASE-5974

 Long running scan produces OutOfOrderScannerNextException
 -

 Key: HBASE-11295
 URL: https://issues.apache.org/jira/browse/HBASE-11295
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.96.0
Reporter: Jeff Cunningham
 Attachments: OutOfOrderScannerNextException.tar.gz


 Attached Files:
 HRegionServer.java - instramented from 0.96.1.1-cdh5.0.0
 HBaseLeaseTimeoutIT.java - reproducing JUnit 4 test
 WaitFilter.java - Scan filter (extends FilterBase) that overrides 
 filterRowKey() to sleep during invocation
 SpliceFilter.proto - Protobuf defintiion for WaitFilter.java
 OutOfOrderScann_InstramentedServer.log - instramented server log
 Steps.txt - this note
 Set up:
 In HBaseLeaseTimeoutIT, create a scan, set the given filter (which sleeps in 
 overridden filterRowKey() method) and set it on the scan, and scan the table.
 This is done in test client_0x0_server_15x10().
 Here's what I'm seeing (see also attached log):
 A new request comes into server (ID 1940798815214593802 - 
 RpcServer.handler=96) and a RegionScanner is created for it, cached by ID, 
 immediately looked up again and cached RegionScannerHolder's nextCallSeq 
 incremeted (now at 1).
 The RegionScan thread goes to sleep in WaitFilter#filterRowKey().
 A short (variable) period later, another request comes into the server (ID 
 8946109289649235722 - RpcServer.handler=98) and the same series of events 
 happen to this request.
 At this point both RegionScanner threads are sleeping in 
 WaitFilter.filterRowKey(). After another period, the client retries another 
 scan request which thinks its next_call_seq is 0.  However, HRegionServer's 
 cached RegionScannerHolder thinks the matching RegionScanner's nextCallSeq 
 should be 1.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11280) Document distributed log replay and distributed log splitting

[
https://issues.apache.org/jira/browse/HBASE-11280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017752#comment-14017752
]

stack commented on HBASE-11280:
---

This is great (I like this kind of stuff 'firsttermWrite Ahead Log
(WAL)/firstterm records)

Location looks correct to me.

Change this The WAL implementation used by HBase is because the HLog you
point to is only an Interface. Implementations implement this Interface.
Currently there is one only but attempts have been made at doing others and we
may be doing a new one soon to do multiwal.

Change this One HLog instance exists ... to be instead Usually there is one
instance of a WAL only per RegionServer... (there is one HLog implementatoin
true but I think you are trying to say one WAL here). We intend there to be
many in the hopefully not too distant future.

WALs used to be kept under filename/hbase/.logs//filename in and before
0.94. Now they are in filename/hbase/WALs/filename

The description of '+titleWAL Splitting, Step by Step/title'
applies to Distributed Log Splitting. You might want to call it out as so?

Maybe mention that the HMaster enrolls all regionservers in the log splitting
process farming log splitting process to the regionservers.

That is what I have so far (about half way through review)

Document distributed log replay and distributed log splitting
-

Key: HBASE-11280
URL: https://issues.apache.org/jira/browse/HBASE-11280
Project: HBase
Issue Type: Sub-task
Components: documentation, MTTR, wal
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
Fix For: 0.99.0

Attachments: HBASE-11280.patch

Enable 'distributed log replay' by default. Depends on hfilev3 being enabled.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11293) Master and Region servers fail to start when hbase.master.ipc.address=0.0.0.0, hbase.regionserver.ipc.address=0.0.0.0 and Kerberos is enabled

2014-06-04 Thread Michael Harp (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017799#comment-14017799
 ] 

Michael Harp commented on HBASE-11293:
--

Yes, we have multiple nics on all hosts.

 Master and Region servers fail to start when 
 hbase.master.ipc.address=0.0.0.0, hbase.regionserver.ipc.address=0.0.0.0 and 
 Kerberos is enabled
 -

 Key: HBASE-11293
 URL: https://issues.apache.org/jira/browse/HBASE-11293
 Project: HBase
  Issue Type: Bug
Reporter: Michael Harp

 Setting 
 {code}
 hbase.master.ipc.address=0.0.0.0
 hbase.regionserver.ipc.address=0.0.0.0
 {code}
 causes the _HOST substitution in hbase/_h...@example.com to result in 
 hbase/0:0:0:0:0:0:0:0...@example.com which in turn causes kerberos 
 authentication to fail.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11274) More general single-row Condition Mutation

2014-06-04 Thread Jean-Marc Spaggiari (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017801#comment-14017801
 ] 

Jean-Marc Spaggiari commented on HBASE-11274:
-

I like the idea. I definitively see some usecases for that in already existing 
big HBase users. Don't we already have a way to order filters with ANDs and 
ORs? So this kind of logic might already be there somewhere and we might be 
able to reuse it?

 More general single-row Condition Mutation
 --

 Key: HBASE-11274
 URL: https://issues.apache.org/jira/browse/HBASE-11274
 Project: HBase
  Issue Type: Improvement
Reporter: Liu Shaohui
Priority: Minor

 Currently, the checkAndDelete and checkAndPut interface  only support atomic 
 mutation with single condition. But in actual apps, we need more general 
 condition-mutation that support multi conditions and logical expression with 
 those conditions.
 For example, to support the following sql
 {quote}
   insert row  where (column A == 'X' and column B == 'Y') or (column C == 'z')
 {quote}
 Suggestions are welcomed.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11295) Long running scan produces OutOfOrderScannerNextException


[ 
https://issues.apache.org/jira/browse/HBASE-11295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017816#comment-14017816
 ] 

stack commented on HBASE-11295:
---

Any chance of your taking a look [~anoop.hbase] ? (it is a bad time for you 
just now...)

 Long running scan produces OutOfOrderScannerNextException
 -

 Key: HBASE-11295
 URL: https://issues.apache.org/jira/browse/HBASE-11295
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.96.0
Reporter: Jeff Cunningham
 Attachments: OutOfOrderScannerNextException.tar.gz


 Attached Files:
 HRegionServer.java - instramented from 0.96.1.1-cdh5.0.0
 HBaseLeaseTimeoutIT.java - reproducing JUnit 4 test
 WaitFilter.java - Scan filter (extends FilterBase) that overrides 
 filterRowKey() to sleep during invocation
 SpliceFilter.proto - Protobuf defintiion for WaitFilter.java
 OutOfOrderScann_InstramentedServer.log - instramented server log
 Steps.txt - this note
 Set up:
 In HBaseLeaseTimeoutIT, create a scan, set the given filter (which sleeps in 
 overridden filterRowKey() method) and set it on the scan, and scan the table.
 This is done in test client_0x0_server_15x10().
 Here's what I'm seeing (see also attached log):
 A new request comes into server (ID 1940798815214593802 - 
 RpcServer.handler=96) and a RegionScanner is created for it, cached by ID, 
 immediately looked up again and cached RegionScannerHolder's nextCallSeq 
 incremeted (now at 1).
 The RegionScan thread goes to sleep in WaitFilter#filterRowKey().
 A short (variable) period later, another request comes into the server (ID 
 8946109289649235722 - RpcServer.handler=98) and the same series of events 
 happen to this request.
 At this point both RegionScanner threads are sleeping in 
 WaitFilter.filterRowKey(). After another period, the client retries another 
 scan request which thinks its next_call_seq is 0.  However, HRegionServer's 
 cached RegionScannerHolder thinks the matching RegionScanner's nextCallSeq 
 should be 1.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11295) Long running scan produces OutOfOrderScannerNextException

2014-06-04 Thread Anoop Sam John (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-11295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017817#comment-14017817
]

Anoop Sam John commented on HBASE-11295:

As in Filte waiting happens, client times out and do retry. If this retry is
allowed, we will fail returning some rows. That is why we added the next seq
no#. In this case throwing exception is desired behaviour. On seeing this
exception the client will restart with the initial request . U can see thos
logs? If ur filter logic is going to be time taking, increase the client
timeout.

Long running scan produces OutOfOrderScannerNextException
-

Key: HBASE-11295
URL: https://issues.apache.org/jira/browse/HBASE-11295
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.96.0
Reporter: Jeff Cunningham
Attachments: OutOfOrderScannerNextException.tar.gz

Attached Files:
HRegionServer.java - instramented from 0.96.1.1-cdh5.0.0
HBaseLeaseTimeoutIT.java - reproducing JUnit 4 test
WaitFilter.java - Scan filter (extends FilterBase) that overrides
filterRowKey() to sleep during invocation
SpliceFilter.proto - Protobuf defintiion for WaitFilter.java
OutOfOrderScann_InstramentedServer.log - instramented server log
Steps.txt - this note
Set up:
In HBaseLeaseTimeoutIT, create a scan, set the given filter (which sleeps in
overridden filterRowKey() method) and set it on the scan, and scan the table.
This is done in test client_0x0_server_15x10().
Here's what I'm seeing (see also attached log):
A new request comes into server (ID 1940798815214593802 -
RpcServer.handler=96) and a RegionScanner is created for it, cached by ID,
immediately looked up again and cached RegionScannerHolder's nextCallSeq
incremeted (now at 1).
The RegionScan thread goes to sleep in WaitFilter#filterRowKey().
A short (variable) period later, another request comes into the server (ID
8946109289649235722 - RpcServer.handler=98) and the same series of events
happen to this request.
At this point both RegionScanner threads are sleeping in
WaitFilter.filterRowKey(). After another period, the client retries another
scan request which thinks its next_call_seq is 0. However, HRegionServer's
cached RegionScannerHolder thinks the matching RegionScanner's nextCallSeq
should be 1.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10289) Avoid random port usage by default JMX Server. Create Custome JMX server


[ 
https://issues.apache.org/jira/browse/HBASE-10289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017819#comment-14017819
 ] 

stack commented on HBASE-10289:
---

[~tianq] Mind making a distinct addendum for trunk that has in it only the doc 
edit Andy suggest?  Thanks.

 Avoid random port usage by default JMX Server. Create Custome JMX server
 

 Key: HBASE-10289
 URL: https://issues.apache.org/jira/browse/HBASE-10289
 Project: HBase
  Issue Type: Improvement
Reporter: nijel
Assignee: Qiang Tian
Priority: Minor
  Labels: stack
 Fix For: 0.99.0

 Attachments: HBASE-10289-v4.patch, HBASE-10289.patch, 
 HBASE-10289_1.patch, HBASE-10289_2.patch, HBASE-10289_3.patch, 
 HBase10289-master.patch, hbase10289-master-v1.patch, 
 hbase10289-master-v2.patch


 If we enable JMX MBean server for HMaster or Region server  through VM 
 arguments, the process will use one random which we cannot configure.
 This can be a problem if that random port is configured for some other 
 service.
 This issue can be avoided by supporting  a custom JMX Server.
 The ports can be configured. If there is no ports configured, it will 
 continue the same way as now.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10856) Prep for 1.0


[ 
https://issues.apache.org/jira/browse/HBASE-10856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017821#comment-14017821
 ] 

stack commented on HBASE-10856:
---

Consider this for 1.0 (caveat the question by Ishan on the end of the issue)

 Prep for 1.0
 

 Key: HBASE-10856
 URL: https://issues.apache.org/jira/browse/HBASE-10856
 Project: HBase
  Issue Type: Umbrella
Reporter: stack
 Fix For: 0.99.0


 Tasks for 1.0 copied here from our '1.0.0' mailing list discussion.  Idea is 
 to file subtasks off this one.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11263) Share the open/close store file thread pool for all store in a region


[ 
https://issues.apache.org/jira/browse/HBASE-11263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017827#comment-14017827
 ] 

stack commented on HBASE-11263:
---

Ok.  +1 then.  Work for a thread pool at server level would be in a different 
JIRA I presume.

 Share the open/close store file thread pool for all store in a region
 -

 Key: HBASE-11263
 URL: https://issues.apache.org/jira/browse/HBASE-11263
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Affects Versions: 0.99.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
 Attachments: HBASE-11263-trunk-v1.diff


 Currently, the open/close store file thread pool is divided equally to all 
 stores of a region. 
 {code}
   protected ThreadPoolExecutor getStoreFileOpenAndCloseThreadPool(
   final String threadNamePrefix) {
 int numStores = Math.max(1, this.htableDescriptor.getFamilies().size());
 int maxThreads = Math.max(1,
 conf.getInt(HConstants.HSTORE_OPEN_AND_CLOSE_THREADS_MAX,
 HConstants.DEFAULT_HSTORE_OPEN_AND_CLOSE_THREADS_MAX)
 / numStores);
 return getOpenAndCloseThreadPool(maxThreads, threadNamePrefix);
   }
 {code}
 This is not very optimal in following scenarios:
 # The data of some column families are very large and there are many hfiles 
 in those stores, and others may be very small and in-memory column families. 
 # Usually we preserve some column families for later needs. The thread pool 
 for these column families are wasted。
 The simple way is to share a big thread pool for all stores to open/close 
 hfiles.  
 Suggestions are welcomed. 
 Thanks. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11297) Remove some synchros in the rpcServer responder


 [ 
https://issues.apache.org/jira/browse/HBASE-11297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Liochon updated HBASE-11297:


Attachment: 11297.v1.patch

 Remove some synchros in the rpcServer responder
 ---

 Key: HBASE-11297
 URL: https://issues.apache.org/jira/browse/HBASE-11297
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.99.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.99.0

 Attachments: 11297.v1.patch


 This is on top of another patch that I'm going to put into another jira.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11297) Remove some synchros in the rpcServer responder


[ 
https://issues.apache.org/jira/browse/HBASE-11297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017845#comment-14017845
 ] 

Nicolas Liochon commented on HBASE-11297:
-

v1. Tests are in progress.

 Remove some synchros in the rpcServer responder
 ---

 Key: HBASE-11297
 URL: https://issues.apache.org/jira/browse/HBASE-11297
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.99.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.99.0

 Attachments: 11297.v1.patch


 This is on top of another patch that I'm going to put into another jira.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11292) Add an undelete operation

2014-06-04 Thread Gary Helmling (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-11292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017841#comment-14017841
]

Gary Helmling commented on HBASE-11292:
---

bq. Would every type of Delete need a corresponding Undelete?

Yes, I think it would be necessary to have complete parity between the two,
since the goal is to cancel any type of delete operation.

{quote}
When KEEP_DELETED_CELLS is on, assume:
* Put is done at t1
* Delete is done at t5
* Undelete is done at t7
If a client does a read at t6, the Delete would be in effect, while if a client
does a read at t8, the Delete would not be in effect. Is that how it'd work?
{quote}

Yes, I think this would work exactly as you describe. In the case of
supporting timestamp-overriding transactions, though, the Delete and Undelete
would occur at the same timestamp (t5), since the Undelete would be rolling
back the Delete that was part of the transaction.

Add an undelete operation
---

Key: HBASE-11292
URL: https://issues.apache.org/jira/browse/HBASE-11292
Project: HBase
Issue Type: New Feature
Components: Deletes
Reporter: Gary Helmling
Labels: Phoenix

While column families can be configured to keep deleted cells (allowing time
range queries to still retrieve those cells), deletes are still somewhat
unique in that they are irreversible operations. Once a delete has been
issued on a cell, the only way to undelete it is to rewrite the data with a
timestamp newer than the delete.
The idea here is to add an undelete operation, that would make it possible
to cancel a previous delete. An undelete operation will be similar to a
delete, in that it will be written as a marker (tombstone doesn't seem like
the right word). The undelete marker, however, will sort prior to a delete
marker, canceling the effect of any following delete.
In the absence of a column family configured to KEEP_DELETED_CELLS, we can't
be sure if a prior delete marker and the effected cells have already been
garbage collected. In this case (column family not configured with
KEEP_DELETED_CELLS) it may be necessary for the server to reject undelete
operations to avoid creating the appearance of a client contact for undeletes
that can't reliably be honored.
I think there are additional subtleties of the implementation to be worked
out, but I'm also interested in a broader discussion of interest in this
capability.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HBASE-11297) Remove some synchros in the rpcServer responder

Nicolas Liochon created HBASE-11297:
---

 Summary: Remove some synchros in the rpcServer responder
 Key: HBASE-11297
 URL: https://issues.apache.org/jira/browse/HBASE-11297
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.99.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.99.0
 Attachments: 11297.v1.patch

This is on top of another patch that I'm going to put into another jira.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10092) Move up on to log4j2

[
https://issues.apache.org/jira/browse/HBASE-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017846#comment-14017846
]

stack commented on HBASE-10092:
---

How to progress on this [~andrew.purt...@gmail.com]?

+ Need to make tgzs and see that stuff lands in right place. Add some excludes
for transitive includes of old log4js.
+ Need to try in standalone, pseudo, and cluster and bring up shell to make
sure logs are ending up in right place?

I'd be tempted to just commit something that basically worked and then work out
the rest in subsequent issues.

Move up on to log4j2

Key: HBASE-10092
URL: https://issues.apache.org/jira/browse/HBASE-10092
Project: HBase
Issue Type: Sub-task
Reporter: stack
Assignee: stack
Attachments: 10092.txt, 10092v2.txt, HBASE-10092.patch

Allows logging with less friction. See http://logging.apache.org/log4j/2.x/
This rather radical transition can be done w/ minor change given they have an
adapter for apache's logging, the one we use. They also have and adapter for
slf4j so we likely can remove at least some of the 4 versions of this module
our dependencies make use of.
I made a start in attached patch but am currently stuck in maven dependency
resolve hell courtesy of our slf4j. Fixing will take some concentration and
a good net connection, an item I currently lack. Other TODOs are that will
need to fix our little log level setting jsp page -- will likely have to undo
our use of hadoop's tool here -- and the config system changes a little.
I will return to this project soon. Will bring numbers.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11297) Remove some synchros in the rpcServer responder


 [ 
https://issues.apache.org/jira/browse/HBASE-11297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Liochon updated HBASE-11297:


Status: Patch Available  (was: Open)

 Remove some synchros in the rpcServer responder
 ---

 Key: HBASE-11297
 URL: https://issues.apache.org/jira/browse/HBASE-11297
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.99.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.99.0

 Attachments: 11297.v1.patch, 11298.v1.patch


 This is on top of another patch that I'm going to put into another jira.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11297) Remove some synchros in the rpcServer responder


 [ 
https://issues.apache.org/jira/browse/HBASE-11297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Liochon updated HBASE-11297:


Attachment: 11298.v1.patch

 Remove some synchros in the rpcServer responder
 ---

 Key: HBASE-11297
 URL: https://issues.apache.org/jira/browse/HBASE-11297
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.99.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.99.0

 Attachments: 11297.v1.patch, 11298.v1.patch


 This is on top of another patch that I'm going to put into another jira.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HBASE-11298) Simplification in RpcServer code

Nicolas Liochon created HBASE-11298:
---

 Summary: Simplification in RpcServer code
 Key: HBASE-11298
 URL: https://issues.apache.org/jira/browse/HBASE-11298
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.99.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.99.0


No important changes here, but I'm doing some other changes on top of this 
(typically HBASE-11297)
 
Note that I've changed the logs, they now belong to the real class instead of 
hijacking Hadoop. I suppose it was historical, but it was as well very 
confusing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11297) Remove some synchros in the rpcServer responder


 [ 
https://issues.apache.org/jira/browse/HBASE-11297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Liochon updated HBASE-11297:


Status: Open  (was: Patch Available)

 Remove some synchros in the rpcServer responder
 ---

 Key: HBASE-11297
 URL: https://issues.apache.org/jira/browse/HBASE-11297
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.99.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.99.0

 Attachments: 11297.v1.patch, 11298.v1.patch


 This is on top of another patch that I'm going to put into another jira.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11298) Simplification in RpcServer code


 [ 
https://issues.apache.org/jira/browse/HBASE-11298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Liochon updated HBASE-11298:


Status: Patch Available  (was: Open)

 Simplification in RpcServer code
 

 Key: HBASE-11298
 URL: https://issues.apache.org/jira/browse/HBASE-11298
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.99.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.99.0

 Attachments: 11298.v1.patch


 No important changes here, but I'm doing some other changes on top of this 
 (typically HBASE-11297)
  
 Note that I've changed the logs, they now belong to the real class instead 
 of hijacking Hadoop. I suppose it was historical, but it was as well very 
 confusing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11297) Remove some synchros in the rpcServer responder


 [ 
https://issues.apache.org/jira/browse/HBASE-11297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Liochon updated HBASE-11297:


Attachment: (was: 11298.v1.patch)

 Remove some synchros in the rpcServer responder
 ---

 Key: HBASE-11297
 URL: https://issues.apache.org/jira/browse/HBASE-11297
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.99.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.99.0

 Attachments: 11297.v1.patch


 This is on top of another patch that I'm going to put into another jira.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11298) Simplification in RpcServer code


 [ 
https://issues.apache.org/jira/browse/HBASE-11298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Liochon updated HBASE-11298:


Attachment: 11298.v1.patch

 Simplification in RpcServer code
 

 Key: HBASE-11298
 URL: https://issues.apache.org/jira/browse/HBASE-11298
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.99.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.99.0

 Attachments: 11298.v1.patch


 No important changes here, but I'm doing some other changes on top of this 
 (typically HBASE-11297)
  
 Note that I've changed the logs, they now belong to the real class instead 
 of hijacking Hadoop. I suppose it was historical, but it was as well very 
 confusing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10092) Move up on to log4j2

[
https://issues.apache.org/jira/browse/HBASE-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017871#comment-14017871
]

Andrew Purtell commented on HBASE-10092:

bq. Need to make tgzs and see that stuff lands in right place. Add some
excludes for transitive includes of old log4js.

I did this and checked mvn dependency:tree. It looks good. I then had to add
back log4j 1 in test scope so we could actually instantiate Hadoop classes for
running unit tests, not sure what that is going to do in an assembly.

bq. Need to try in standalone, pseudo, and cluster and bring up shell to make
sure logs are ending up in right place

Yes, the new log4j2.xml files are untested and I'm sure need work.

Move up on to log4j2

Key: HBASE-10092
URL: https://issues.apache.org/jira/browse/HBASE-10092
Project: HBase
Issue Type: Sub-task
Reporter: stack
Assignee: stack
Attachments: 10092.txt, 10092v2.txt, HBASE-10092.patch

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Comment Edited] (HBASE-10092) Move up on to log4j2

[
https://issues.apache.org/jira/browse/HBASE-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017871#comment-14017871
]

Andrew Purtell edited comment on HBASE-10092 at 6/4/14 5:10 PM:

bq. Need to make tgzs and see that stuff lands in right place. Add some
excludes for transitive includes of old log4js.

I did this and checked mvn dependency:tree. It looks good. I then had to add
back log4j 1 and slf4j in test scope so we could actually instantiate Hadoop
classes for running unit tests (Java logging is a disaster), not sure what that
is going to do in an assembly.

bq. Need to try in standalone, pseudo, and cluster and bring up shell to make
sure logs are ending up in right place

Yes, the new log4j2.xml files are untested and I'm sure need work.

was (Author: apurtell):
bq. Need to make tgzs and see that stuff lands in right place. Add some
excludes for transitive includes of old log4js.

bq. Need to try in standalone, pseudo, and cluster and bring up shell to make
sure logs are ending up in right place

Yes, the new log4j2.xml files are untested and I'm sure need work.

Move up on to log4j2

Key: HBASE-10092
URL: https://issues.apache.org/jira/browse/HBASE-10092
Project: HBase
Issue Type: Sub-task
Reporter: stack
Assignee: stack
Attachments: 10092.txt, 10092v2.txt, HBASE-10092.patch

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11298) Simplification in RpcServer code


[ 
https://issues.apache.org/jira/browse/HBASE-11298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017887#comment-14017887
 ] 

Andrew Purtell commented on HBASE-11298:


+1

I would leave this hunk out of a 0.98 patch:
{code}
--- hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
+++ hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
@@ -144,9 +144,7 @@ import com.google.protobuf.TextFormat;
  */
 @InterfaceAudience.Private
 public class RpcServer implements RpcServerInterface {
-  // The logging package is deliberately outside of standard o.a.h.h package 
so it is not on
-  // by default.
-  public static final Log LOG = 
LogFactory.getLog(org.apache.hadoop.ipc.RpcServer);
+  public static final Log LOG = LogFactory.getLog(RpcServer.class);
 
   private final boolean authorize;
   private boolean isSecurityEnabled;
{code}

 Simplification in RpcServer code
 

 Key: HBASE-11298
 URL: https://issues.apache.org/jira/browse/HBASE-11298
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.99.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.99.0

 Attachments: 11298.v1.patch


 No important changes here, but I'm doing some other changes on top of this 
 (typically HBASE-11297)
  
 Note that I've changed the logs, they now belong to the real class instead 
 of hijacking Hadoop. I suppose it was historical, but it was as well very 
 confusing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11292) Add an undelete operation

2014-06-04 Thread Gary Helmling (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-11292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017885#comment-14017885
]

Gary Helmling commented on HBASE-11292:
---

bq. we have delete bloom filters to avoid seeks to the beginning on the row to
check for family delete markers. Those would no longer work, or in other words
we'd need boom filters for the undelete markers.

Hmm, I wasn't familiar with these, but, yes, sounds like we would unfortunately
need another set of bloom filters for the undeletes. There would also need to
be some change to the use of the delete bloom filters if operations were
ordered by seqid, wouldn't there?

bq. will we every need so to undo an undelete?

Yes, I thought of this as well. What undoes the undeletes? In a sense this
shifts the irreversible operation to the undelete. I think this is an inherent
problem with the approach of sorting operations within the same timestamp by
operation type. There is always going to be something that sorts first, which
effectively becomes undoable.

bq. how does this fit into the discussion about using sequence numbers to order
operations?

I think the discussion to order operations by seqid is simply an alternate
approach to the some of the same underlying problems. For the issue mentioned
above, sorting by seqid is conceptually simpler -- there is no need for a
separate undelete operation and undoing a delete is possible by re-issuing a
new put for the previous value at the same timestamp as the previous delete.
Of course that means that the final outcome depends on the server observed
ordering of the operations. But at the same time, there is no irreversible
operation.

For the use case I'm considering (a single client rolling back it's previously
persisted changes at a given timestamp), ordering by seqid would also work.
Since it's a single client rolling back it's own operations, the operations can
be ordered by the client, so there's no lack of determinism in server side
ordering. Rolling back a delete with seqid ordering would be slightly more
complicated, though, since the client would have to perform a read to find the
previous value prior to the delete, then issue a new put, instead of simply
issuing an undelete with the same parameters as the prior delete.

Or you could combine both approaches and order operations by seqid, but add an
undelete operation as well. The undelete would then sort prior to the delete
by virtue of being issued after it. It would still be a no-read operation.
And the undelete could be undone by issuing a new delete. Maybe this
combination ultimately winds up being best.

Thanks for the comments. I'm open to however we can most efficiently solve
this with the least amount of added complexity.

Add an undelete operation
---

Key: HBASE-11292
URL: https://issues.apache.org/jira/browse/HBASE-11292
Project: HBase
Issue Type: New Feature
Components: Deletes
Reporter: Gary Helmling
Labels: Phoenix

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11298) Simplification in RpcServer code


[ 
https://issues.apache.org/jira/browse/HBASE-11298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017884#comment-14017884
 ] 

Andrew Purtell commented on HBASE-11298:


Please consider 0.98 also. Or I could do a back port.

 Simplification in RpcServer code
 

 Key: HBASE-11298
 URL: https://issues.apache.org/jira/browse/HBASE-11298
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.99.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.99.0

 Attachments: 11298.v1.patch


 No important changes here, but I'm doing some other changes on top of this 
 (typically HBASE-11297)
  
 Note that I've changed the logs, they now belong to the real class instead 
 of hijacking Hadoop. I suppose it was historical, but it was as well very 
 confusing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11297) Remove some synchros in the rpcServer responder


[ 
https://issues.apache.org/jira/browse/HBASE-11297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017881#comment-14017881
 ] 

Andrew Purtell commented on HBASE-11297:


Please consider 0.98 also. Or I could do a back port.

 Remove some synchros in the rpcServer responder
 ---

 Key: HBASE-11297
 URL: https://issues.apache.org/jira/browse/HBASE-11297
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.99.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.99.0

 Attachments: 11297.v1.patch


 This is on top of another patch that I'm going to put into another jira.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11297) Remove some synchros in the rpcServer responder


[ 
https://issues.apache.org/jira/browse/HBASE-11297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017889#comment-14017889
 ] 

Andrew Purtell commented on HBASE-11297:


Let me try HBASE-11298 and this under load

 Remove some synchros in the rpcServer responder
 ---

 Key: HBASE-11297
 URL: https://issues.apache.org/jira/browse/HBASE-11297
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.99.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.99.0

 Attachments: 11297.v1.patch


 This is on top of another patch that I'm going to put into another jira.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11298) Simplification in RpcServer code


[ 
https://issues.apache.org/jira/browse/HBASE-11298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017890#comment-14017890
 ] 

stack commented on HBASE-11298:
---

bq. Note that I've changed the logs, they now belong to the real class 
instead of hijacking Hadoop.

RPC DEBUG is verbose, more verbose than all other hbase DEBUG logging so much 
so that it overwhelms.  Above is an old means of enabling RPC DEBUG independent 
of DEBUG level everywhere else in HBASE.  We'd doc'd it here: 
http://hbase.apache.org/book.html#rpc.logging

Maybe not INFO level is default and RPC logging has had an edit, it might be OK 
making it so DEBUG enable'd RPC logging. have you checked?

Oh, you made the RPC trace-level?  Thats better (include doc change in your 
patch?)

Why we no longer need this: -if (inHandler) {

Should you pass in the buffer readPreamble works on rather than rely on 'this'? 
 I don't like methods that have side effects; i.e. setting 'this' items if it 
returns successufully. On other hand this new method sets the buffer and 
authmethod... so maybe just leave as is and doc it sets these two data members 
on success.

Otherwise, cleanup looks good to me.  +1 to commit if hadoopqa doesn't burp.



 Simplification in RpcServer code
 

 Key: HBASE-11298
 URL: https://issues.apache.org/jira/browse/HBASE-11298
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.99.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.99.0

 Attachments: 11298.v1.patch


 No important changes here, but I'm doing some other changes on top of this 
 (typically HBASE-11297)
  
 Note that I've changed the logs, they now belong to the real class instead 
 of hijacking Hadoop. I suppose it was historical, but it was as well very 
 confusing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11298) Simplification in RpcServer code


[ 
https://issues.apache.org/jira/browse/HBASE-11298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017891#comment-14017891
 ] 

Nicolas Liochon commented on HBASE-11298:
-

It's ok I can do the (partial)backport as you're ok for 98.


 Simplification in RpcServer code
 

 Key: HBASE-11298
 URL: https://issues.apache.org/jira/browse/HBASE-11298
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.99.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.99.0

 Attachments: 11298.v1.patch


 No important changes here, but I'm doing some other changes on top of this 
 (typically HBASE-11297)
  
 Note that I've changed the logs, they now belong to the real class instead 
 of hijacking Hadoop. I suppose it was historical, but it was as well very 
 confusing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11292) Add an undelete operation

[
https://issues.apache.org/jira/browse/HBASE-11292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017894#comment-14017894
]

stack commented on HBASE-11292:
---

bq. There is always going to be something that sorts first, which effectively
becomes undoable.

Our sorting on type is a mistake (as [~sershe] has noted a few times). Would
fixing this help here?

Add an undelete operation
---

Key: HBASE-11292
URL: https://issues.apache.org/jira/browse/HBASE-11292
Project: HBase
Issue Type: New Feature
Components: Deletes
Reporter: Gary Helmling
Labels: Phoenix

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11094) Distributed log replay is incompatible for rolling restarts

2014-06-04 Thread Jeffrey Zhong (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jeffrey Zhong updated HBASE-11094:
--

Attachment: hbase-11094-v5.patch

The line length 100 warning is from protobuf generated files. The
TestLogRollingNoCluster doesn't fail in my env. The v5 addressed a test failure
from my local full test suite run. Thanks.

Distributed log replay is incompatible for rolling restarts
---

Key: HBASE-11094
URL: https://issues.apache.org/jira/browse/HBASE-11094
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Jeffrey Zhong
Priority: Blocker
Fix For: 0.99.0

Attachments: hbase-11094-v2.patch, hbase-11094-v3.patch,
hbase-11094-v4.patch, hbase-11094-v5.patch, hbase-11094.patch

0.99.0 comes with dist log replay by default (HBASE-10888). However, reading
the code and discussing this with Jeffrey, we realized that the dist log
replay code is not compatible with rolling upgrades from 0.98.0 and 1.0.0.
The issue is that, the region server looks at it own configuration to decide
whether the region should be opened in replay mode or not. The open region
RPC does not contain that info. So if dist log replay is enabled on master,
the master will assign the region and schedule replay tasks. If the region is
opened in a RS that does not have this conf enabled, then it will happily
open the region in normal mode (not replay mode) causing possible (transient)
data loss.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11298) Simplification in RpcServer code


[ 
https://issues.apache.org/jira/browse/HBASE-11298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017897#comment-14017897
 ] 

Andrew Purtell commented on HBASE-11298:


bq. It's ok I can do the (partial)backport as you're ok for 98.

Thanks. I'll wait for the final version rather than backport myself to try out 
HBASE-11297 and other related changes. However as soon as we have this issue 
committed in 0.98 I will gladly do that. I am about to start tracking down 
connection starvation issues under high load. I don't want to do that only to 
find results obviated by cleanups. :-)

 Simplification in RpcServer code
 

 Key: HBASE-11298
 URL: https://issues.apache.org/jira/browse/HBASE-11298
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.99.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.99.0

 Attachments: 11298.v1.patch


 No important changes here, but I'm doing some other changes on top of this 
 (typically HBASE-11297)
  
 Note that I've changed the logs, they now belong to the real class instead 
 of hijacking Hadoop. I suppose it was historical, but it was as well very 
 confusing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11297) Remove some synchros in the rpcServer responder


[ 
https://issues.apache.org/jira/browse/HBASE-11297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017902#comment-14017902
 ] 

stack commented on HBASE-11297:
---

Can you write up high-level what has been changed?  There is code movement and 
I see the queue now is unsynchronized -- is this going to be safe? -- but what 
else?  A few high level notes would help us review.  Thanks [~nkeywal]

 Remove some synchros in the rpcServer responder
 ---

 Key: HBASE-11297
 URL: https://issues.apache.org/jira/browse/HBASE-11297
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.99.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.99.0

 Attachments: 11297.v1.patch


 This is on top of another patch that I'm going to put into another jira.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-6712) Implement checkAndIncrement

2014-06-04 Thread Jean-Marc Spaggiari (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017904#comment-14017904
 ] 

Jean-Marc Spaggiari commented on HBASE-6712:


Do we still have any driver for this issue? Or should we resolve as Later?

 Implement checkAndIncrement
 ---

 Key: HBASE-6712
 URL: https://issues.apache.org/jira/browse/HBASE-6712
 Project: HBase
  Issue Type: New Feature
  Components: Client
Affects Versions: 0.92.1
Reporter: Stefan Baritchii
Assignee: Ted Yu

 increment should throw an exception if a row does not exist. instead it 
 creates the row. checkAndIncrement may be also a solution to it but needs 
 development.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Reopened] (HBASE-10289) Avoid random port usage by default JMX Server. Create Custome JMX server


 [ 
https://issues.apache.org/jira/browse/HBASE-10289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell reopened HBASE-10289:



Reopening for 0.98 patch and trunk doc addendum. Thanks [~tianq]

 Avoid random port usage by default JMX Server. Create Custome JMX server
 

 Key: HBASE-10289
 URL: https://issues.apache.org/jira/browse/HBASE-10289
 Project: HBase
  Issue Type: Improvement
Reporter: nijel
Assignee: Qiang Tian
Priority: Minor
  Labels: stack
 Fix For: 0.99.0

 Attachments: HBASE-10289-v4.patch, HBASE-10289.patch, 
 HBASE-10289_1.patch, HBASE-10289_2.patch, HBASE-10289_3.patch, 
 HBase10289-master.patch, hbase10289-master-v1.patch, 
 hbase10289-master-v2.patch


 If we enable JMX MBean server for HMaster or Region server  through VM 
 arguments, the process will use one random which we cannot configure.
 This can be a problem if that random port is configured for some other 
 service.
 This issue can be avoided by supporting  a custom JMX Server.
 The ports can be configured. If there is no ports configured, it will 
 continue the same way as now.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HBASE-11299) Stop making javadoc for generated classes

stack created HBASE-11299:
-

 Summary: Stop making javadoc for generated classes
 Key: HBASE-11299
 URL: https://issues.apache.org/jira/browse/HBASE-11299
 Project: HBase
  Issue Type: Bug
  Components: documentation
Reporter: stack
Priority: Minor
 Attachments: 11299.txt

Our javadoc generation is taking a long time.  Also, when we go to update our 
site, the javadoc changes are so many -- the date has changed on generated doc 
-- that it takes for ever to commit (it is frighteningly long and noobs can 
only think they've done something wrong).

I have been trying various tricks to get javadoc tool to ignore the generated 
protobuf classes but am not having much success (it is a background process and 
the distractions don't help).  Stashing here changes I have so far to the 
javadoc pom which excludes some of the unneeded files.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11299) Stop making javadoc for generated classes


 [ 
https://issues.apache.org/jira/browse/HBASE-11299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-11299:
--

Attachment: 11299.txt

 Stop making javadoc for generated classes
 -

 Key: HBASE-11299
 URL: https://issues.apache.org/jira/browse/HBASE-11299
 Project: HBase
  Issue Type: Bug
  Components: documentation
Reporter: stack
Priority: Minor
 Attachments: 11299.txt


 Our javadoc generation is taking a long time.  Also, when we go to update our 
 site, the javadoc changes are so many -- the date has changed on generated 
 doc -- that it takes for ever to commit (it is frighteningly long and noobs 
 can only think they've done something wrong).
 I have been trying various tricks to get javadoc tool to ignore the generated 
 protobuf classes but am not having much success (it is a background process 
 and the distractions don't help).  Stashing here changes I have so far to the 
 javadoc pom which excludes some of the unneeded files.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10646) Enable security features by default for 1.0


[ 
https://issues.apache.org/jira/browse/HBASE-10646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017914#comment-14017914
 ] 

Andrew Purtell commented on HBASE-10646:


This JIRA proposes automatically setting up the more complicated piecemeal 
configurations of security components programmatically based on a single 
boolean toggle. There would be no more or less overhead than before, it's 
configuration only changes. There is another JIRA open that considers moving 
security features from coprocessors into core, that is .HBASE-11127. Your 
concerns about additional unconditional overhead in operation processing are 
definitely valid there [~ishanc].

bq. Will main RPCs like Get, Put, etc (apart from the admin RPCs) also be 
secured after that change? 

Only if enabled.

bq. Also, +1 for a simple security = false option.

That is what is proposed on this issue.

 Enable security features by default for 1.0
 ---

 Key: HBASE-10646
 URL: https://issues.apache.org/jira/browse/HBASE-10646
 Project: HBase
  Issue Type: Task
Affects Versions: 0.99.0
Reporter: Andrew Purtell

 As discussed in the last PMC meeting, we should enable security features by 
 default in 1.0.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-10915) Decouple region closing (HM and HRS) from ZK


 [ 
https://issues.apache.org/jira/browse/HBASE-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-10915:
--

   Resolution: Fixed
Fix Version/s: 0.99.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed to master.  Thanks for the patch [~mantonov].  FYI, add a version 
number to your patches so easier to tell them apart...: i.e. the second patch 
should have v2 or 2 appended and so on.  Good stuff.

 Decouple region closing (HM and HRS) from ZK
 

 Key: HBASE-10915
 URL: https://issues.apache.org/jira/browse/HBASE-10915
 Project: HBase
  Issue Type: Sub-task
  Components: Consensus, Zookeeper
Affects Versions: 0.99.0
Reporter: Mikhail Antonov
Assignee: Mikhail Antonov
 Fix For: 0.99.0

 Attachments: HBASE-10915 (2).patch, HBASE-10915.patch, 
 HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, 
 HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, 
 HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, 
 HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch


 Decouple region closing from ZK. 
 Includes RS side (CloseRegionHandler), HM side (ClosedRegionHandler) and the 
 code using (HRegionServer, RSRpcServices etc).
 May need small changes in AssignmentManager.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11127) Move security features into core


[ 
https://issues.apache.org/jira/browse/HBASE-11127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017923#comment-14017923
 ] 

Andrew Purtell commented on HBASE-11127:


On HBASE-10646 and 
https://issues.apache.org/jira/browse/HBASE-10646?focusedCommentId=14017200page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14017200
 [~ishanc] said:
{quote}
Will main RPCs like Get, Put, etc (apart from the admin RPCs) also be secured 
after that change? Any extra overhead in these RPCs would be unacceptable in 
our use case.
{quote}

In the context of the discussion on this issue, the answer I think must be yes. 
We split out the security components and in fact developed the coprocessor 
framework exactly so security would not add overhead in response processing if 
security features were not required. (Strictly speaking each coprocessor hook 
adds ~100ns but that is unavoidable and we take great care to limit the number 
of hook sites in hot code.)

 Move security features into core
 

 Key: HBASE-11127
 URL: https://issues.apache.org/jira/browse/HBASE-11127
 Project: HBase
  Issue Type: Improvement
Reporter: Andrew Purtell

 HBASE-11126 mentions concurrency issues we are running into as the security 
 code increases in sophistication, due to current placement of coprocessor 
 hooks, and proposes a solution to those issues with the expectation that 
 security code remains outside of core in coprocessors. However, as an 
 alternative we could consider moving all AccessController and 
 VisibilityController related code into core. Worth discussing? 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11263) Share the open/close store file thread pool for all store in a region


[ 
https://issues.apache.org/jira/browse/HBASE-11263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017925#comment-14017925
 ] 

Andrew Purtell commented on HBASE-11263:


bq. Ok.  +1 then.  Work for a thread pool at server level would be in a 
different JIRA I presume.

I'd argue not much point for this change if thread pool at server level is 
being worked on now or next.

 Share the open/close store file thread pool for all store in a region
 -

 Key: HBASE-11263
 URL: https://issues.apache.org/jira/browse/HBASE-11263
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Affects Versions: 0.99.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
 Attachments: HBASE-11263-trunk-v1.diff


 Currently, the open/close store file thread pool is divided equally to all 
 stores of a region. 
 {code}
   protected ThreadPoolExecutor getStoreFileOpenAndCloseThreadPool(
   final String threadNamePrefix) {
 int numStores = Math.max(1, this.htableDescriptor.getFamilies().size());
 int maxThreads = Math.max(1,
 conf.getInt(HConstants.HSTORE_OPEN_AND_CLOSE_THREADS_MAX,
 HConstants.DEFAULT_HSTORE_OPEN_AND_CLOSE_THREADS_MAX)
 / numStores);
 return getOpenAndCloseThreadPool(maxThreads, threadNamePrefix);
   }
 {code}
 This is not very optimal in following scenarios:
 # The data of some column families are very large and there are many hfiles 
 in those stores, and others may be very small and in-memory column families. 
 # Usually we preserve some column families for later needs. The thread pool 
 for these column families are wasted。
 The simple way is to share a big thread pool for all stores to open/close 
 hfiles.  
 Suggestions are welcomed. 
 Thanks. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10915) Decouple region closing (HM and HRS) from ZK


[ 
https://issues.apache.org/jira/browse/HBASE-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017929#comment-14017929
 ] 

Mikhail Antonov commented on HBASE-10915:
-

Thanks [~stack]. next time will add version (I thought the other way around - 
when patches are all named the same, JIRA sorts them by timestamp).

 Decouple region closing (HM and HRS) from ZK
 

 Key: HBASE-10915
 URL: https://issues.apache.org/jira/browse/HBASE-10915
 Project: HBase
  Issue Type: Sub-task
  Components: Consensus, Zookeeper
Affects Versions: 0.99.0
Reporter: Mikhail Antonov
Assignee: Mikhail Antonov
 Fix For: 0.99.0

 Attachments: HBASE-10915 (2).patch, HBASE-10915.patch, 
 HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, 
 HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, 
 HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, 
 HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch


 Decouple region closing from ZK. 
 Includes RS side (CloseRegionHandler), HM side (ClosedRegionHandler) and the 
 code using (HRegionServer, RSRpcServices etc).
 May need small changes in AssignmentManager.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11297) Remove some synchros in the rpcServer responder

[
https://issues.apache.org/jira/browse/HBASE-11297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017936#comment-14017936
]

Nicolas Liochon commented on HBASE-11297:
-

The real changes are only on the doRespond path.
There are some changes in purge, but the behavior should not change there: the
documentation was saying that were discarding the calls, but the implementation
was closing the connection. It's not really safe to do something different
anyway: imagine a half-replied call, you can't discard it.

for the doRespond, instead of locking when we wanted to add a call to the list,
we lock only if we want to write on the channel. We also to a tryLock to return
immediately, so the handler does not wait.

The test are still running here (I've done multiple versions, the previous
versions were ok, but...). I haven't yet ran the perf tests. It targets the
scenario with multiple clients on the same connection. It's more or less a
first step for having more Responders (depending on the test results). I will
do another deep review, but I wanted to show what I was working on.

Remove some synchros in the rpcServer responder
---

Key: HBASE-11297
URL: https://issues.apache.org/jira/browse/HBASE-11297
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.99.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Fix For: 0.99.0

Attachments: 11297.v1.patch

This is on top of another patch that I'm going to put into another jira.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-6139) Add troubleshooting section for CentOS 6.2 page allocation failure issue


 [ 
https://issues.apache.org/jira/browse/HBASE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-6139:
-

   Resolution: Fixed
Fix Version/s: 0.99.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed to master.  Thanks [~misty]

 Add troubleshooting section for CentOS 6.2 page allocation failure issue
 

 Key: HBASE-6139
 URL: https://issues.apache.org/jira/browse/HBASE-6139
 Project: HBase
  Issue Type: Task
  Components: documentation
Reporter: Andrew Purtell
Assignee: Misty Stanley-Jones
 Fix For: 0.99.0

 Attachments: HBASE-6139.patch


 Tim Robertson reports:
 bq. HBase CentOS version 6.2 reports kernel: java: page allocation failure. 
 order:4, mode:0x20. Any ideas anyone?
 Then:
 bq. echo 360448  /proc/sys/vm/min_free_kbytes appears to stop page 
 allocation failure using HBase on CentOS 6.2
 If this is the proper fix for this condition, we should document it.
 @Tim, how did you arrive at 360448?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-7912) HBase Backup/Restore Based on HBase Snapshot

2014-06-04 Thread Demai Ni (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017973#comment-14017973
]

Demai Ni commented on HBASE-7912:
-

[~fenghh], many thanks for the comments

{quote}
Use case example 1 in page 3: The full backup doesn't contain data of table3
and table4, so when restoring table3 and table4, their data are all restored
from the incremental backups, right? Sounds it's not a typical
scenario(full-backup + incremental backups) for backup/restore.
{quote}
during step c. .. user adds other table.. this actually triggers an implicite
full backup for table 3 and table 4. So when restore them in the future, the
data will come both full and incremental backup.

{quote}
4. Full Backup: Does log roll take place after taking (full) snapshot? What
if new writes arrive after taking snapshot but before log roll?
{quote}
the logic is to take log roll first and then snapshot. if new writes arrive in
between, it will be saved in the full backup image. And the same writes will be
saved again in the next incremental backup. The approach is to ensure no data
loss by allowing duplicate puts during restore.

{quote}
5. Incremental Backup: What if some RS fails during the log roll procedure so
that not all current log number are recorded onto ZooKeeper?
{quote}
in such case, the backup process will abort, and the clean up logic is the same
as [HBASE-11172 cancel a backup process |
https://issues.apache.org/jira/browse/HBASE-11172]. The code will remove the
incomplete backup image and roll back zookeeper state to the previous backup.

{quote}
What if some log files are archived/deleted between two incremental backups and
are not included in any incremental backup? Is it possible?
{quote}
Good point. (also thanks to [~mbertozzi], who pointed out the same problem
earlier). There is a log cleaner that hasn't been included in the patch yet. It
is called BackupLogCleaner extended from BaseLogCleanerDelegate, as part of
hbase.master.logcleaner.plugins. It would keep the logs. The side-effect would
be (if user don't do incremental too often) too much log files left. We have a
stop -all feature to remove all backup tables, also will free up the logs.

Thanks for pointing out the typo. I will fix them up in the doc.

HBase Backup/Restore Based on HBase Snapshot

A common

[jira] [Commented] (HBASE-11298) Simplification in RpcServer code

[
https://issues.apache.org/jira/browse/HBASE-11298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017988#comment-14017988
]

Hadoop QA commented on HBASE-11298:
---

{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12648353/11298.v1.patch
against trunk revision .
ATTACHMENT ID: 12648353

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100

{color:green}+1 site{color}. The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9689//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9689//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9689//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9689//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9689//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9689//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9689//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9689//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9689//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9689//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9689//console

This message is automatically generated.

Simplification in RpcServer code

Key: HBASE-11298
URL: https://issues.apache.org/jira/browse/HBASE-11298
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.99.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Fix For: 0.99.0

Attachments: 11298.v1.patch

No important changes here, but I'm doing some other changes on top of this
(typically HBASE-11297)

Note that I've changed the logs, they now belong to the real class instead
of hijacking Hadoop. I suppose it was historical, but it was as well very
confusing.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10577) Remove unnecessary looping in FSHLog

[
https://issues.apache.org/jira/browse/HBASE-10577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017997#comment-14017997
]

stack commented on HBASE-10577:
---

Closing as 'wont fix'.

I think we need what we have here because SyncRunners run for the life of the
RS with batches being added to the SyncRunner queue continuously by the Writer
thread; i.e. we are not taking a batch, running, then taking a new batch.

We are being careful we don't go beyond the currently highest sync'd sequence
id. While someone else may have let go of all our handlers after pushing the
highest sequence id beyond our current sync future, there is nothing preventing
the Writer thread adding a new batch of Sync Futures concurrently.

Please reopen if I am not understanding correctly. Thanks.

Remove unnecessary looping in FSHLog

Key: HBASE-10577
URL: https://issues.apache.org/jira/browse/HBASE-10577
Project: HBase
Issue Type: Bug
Components: wal
Affects Versions: 0.99.0
Reporter: Himanshu Vashishtha

In the new disruptor based FSHLog, the Syncer threads are handed a batch of
SyncFuture objects from the RingBufferHandler. The Syncer then invokes a sync
call on the current writer instance.
This handing of batch is done in serially in RingBufferHandler, that is,
every syncer receives a non overlapping batch of SyncFutures. Once synced,
Syncer thread updates highestSyncedSequence.
In the run method of Syncer, we have:
{code}
long currentHighestSyncedSequence = highestSyncedSequence.get();
if (currentSequence currentHighestSyncedSequence) {
syncCount += releaseSyncFuture(takeSyncFuture,
currentHighestSyncedSequence, null);
// Done with the 'take'. Go around again and do a new 'take'.
continue;
}
{code}
I find this logic of polling the BlockingQueue again in this condition
un-necessary. When the currentHighestSyncedSequence is already greater than
currentSequence, then doesn't it mean some other Syncer has already synced
SyncFuture of these ops ? And, we should just go ahead and release all the
SyncFutures for this batch to unblock the handlers. That would avoid polling
the Blockingqueue for all SyncFuture objects in this case.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-10641) Configurable Bucket Sizes in bucketCache

2014-06-04 Thread Nick Dimiduk (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk updated HBASE-10641:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Pushed to 0.98 and master. Thanks for the reviews.

 Configurable Bucket Sizes in bucketCache
 

 Key: HBASE-10641
 URL: https://issues.apache.org/jira/browse/HBASE-10641
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Biju Nair
Assignee: Nick Dimiduk
  Labels: cache
 Fix For: 0.99.0, 0.98.4

 Attachments: HBASE-10641.00.patch, HBASE-10641.01-0.98.patch, 
 HBASE-10641.01.patch, HBASE-10641.02-0.98.patch, HBASE-10641.02.patch


 In the current implementation of BucketCache, the sizes of buckets created 
 for different blocksizes is fixed. Given that the all the blocksizes will not 
 be used, it would be good to have the bucketCache configurable. The block 
 sizes for which buckets need to be created and the size of the buckets for 
 each block size should be configurable.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11295) Long running scan produces OutOfOrderScannerNextException

2014-06-04 Thread Jeff Cunningham (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018135#comment-14018135
 ] 

Jeff Cunningham commented on HBASE-11295:
-

Anoop, thanks for response.

Unfortunately, OutOfOrderScannerNextException that we get when nextCallSeq is 
different is a DoNotRetryIOException, so no retry occurs.  The region server 
log has no record of this event.  We only see this on the (user) client side.

Interestingly, the server's cached nextCallSeq is incremented before the server 
scan is executed.  When client retries, their nextCallSeq values are already 
out of sync -- the server scan is still executing and the client had no 
notification of timeout or error.  Should the server's cached nextCallSeq value 
only be incremented after the scan comes back (or fails)?

If you look at the server log in the attached tarball, you will see the 
sequence for the one client scan (notice the handler threadIDs):
15:15:54,824 (RpcServer.handler=94) -  Getting scanner for new request: 
1940798815214593802
15:15:54,824 (RpcServer.handler=94) -1940798815214593802 nextCallSeq: 0
15:15:54,825 (RpcServer.handler=96) -  Getting scanner for ID: 
1940798815214593802
15:15:54,825 (RpcServer.handler=96) -1940798815214593802 nextCallSeq: 0
15:15:54,825 (RpcServer.handler=96) -  incrementing nexCallSeq for : 
1940798815214593802
15:15:54,825 (RpcServer.handler=96) - WaitFilter snoozin for (15) ms.
15:16:55,135 (RpcServer.handler=97) -  Getting scanner for ID: 
1940798815214593802
15:16:55,135 (RpcServer.handler=97) -1940798815214593802 nextCallSeq: 1
15:16:55,142 (RpcServer.handler=92) -  Getting scanner for new request: 
8946109289649235722
15:16:55,143 (RpcServer.handler=92) -8946109289649235722 nextCallSeq: 0
15:16:55,143 (RpcServer.handler=98) -  Getting scanner for ID: 
8946109289649235722
15:16:55,143 (RpcServer.handler=98) -8946109289649235722 nextCallSeq: 0
15:16:55,143 (RpcServer.handler=98) -  incrementing nexCallSeq for : 
8946109289649235722
15:16:55,143 (RpcServer.handler=98) - WaitFilter snoozin for (15) ms.
 Client sees OutOfOrderScannerException at this point after ~ 1 min. Nothing 
 in region server logs. 



 Long running scan produces OutOfOrderScannerNextException
 -

 Key: HBASE-11295
 URL: https://issues.apache.org/jira/browse/HBASE-11295
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.96.0
Reporter: Jeff Cunningham
 Attachments: OutOfOrderScannerNextException.tar.gz


 Attached Files:
 HRegionServer.java - instramented from 0.96.1.1-cdh5.0.0
 HBaseLeaseTimeoutIT.java - reproducing JUnit 4 test
 WaitFilter.java - Scan filter (extends FilterBase) that overrides 
 filterRowKey() to sleep during invocation
 SpliceFilter.proto - Protobuf defintiion for WaitFilter.java
 OutOfOrderScann_InstramentedServer.log - instramented server log
 Steps.txt - this note
 Set up:
 In HBaseLeaseTimeoutIT, create a scan, set the given filter (which sleeps in 
 overridden filterRowKey() method) and set it on the scan, and scan the table.
 This is done in test client_0x0_server_15x10().
 Here's what I'm seeing (see also attached log):
 A new request comes into server (ID 1940798815214593802 - 
 RpcServer.handler=96) and a RegionScanner is created for it, cached by ID, 
 immediately looked up again and cached RegionScannerHolder's nextCallSeq 
 incremeted (now at 1).
 The RegionScan thread goes to sleep in WaitFilter#filterRowKey().
 A short (variable) period later, another request comes into the server (ID 
 8946109289649235722 - RpcServer.handler=98) and the same series of events 
 happen to this request.
 At this point both RegionScanner threads are sleeping in 
 WaitFilter.filterRowKey(). After another period, the client retries another 
 scan request which thinks its next_call_seq is 0.  However, HRegionServer's 
 cached RegionScannerHolder thinks the matching RegionScanner's nextCallSeq 
 should be 1.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11094) Distributed log replay is incompatible for rolling restarts


[ 
https://issues.apache.org/jira/browse/HBASE-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018153#comment-14018153
 ] 

Hadoop QA commented on HBASE-11094:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12648358/hbase-11094-v5.patch
  against trunk revision .
  ATTACHMENT ID: 12648358

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 29 new 
or modified tests.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+result = result  (hasOpenForDistributedLogReplay() == 
other.hasOpenForDistributedLogReplay());
+  new java.lang.String[] { Region, VersionOfOfflineNode, 
FavoredNodes, OpenForDistributedLogReplay, });

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 

 {color:red}-1 core zombie tests{color}.  There are 1 zombie test(s):   
at 
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnDatanodeDeath(TestLogRolling.java:371)

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9691//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9691//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9691//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9691//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9691//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9691//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9691//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9691//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9691//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9691//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9691//console

This message is automatically generated.

 Distributed log replay is incompatible for rolling restarts
 ---

 Key: HBASE-11094
 URL: https://issues.apache.org/jira/browse/HBASE-11094
 Project: HBase
  Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Jeffrey Zhong
Priority: Blocker
 Fix For: 0.99.0

 Attachments: hbase-11094-v2.patch, hbase-11094-v3.patch, 
 hbase-11094-v4.patch, hbase-11094-v5.patch, hbase-11094.patch


 0.99.0 comes with dist log replay by default (HBASE-10888). However, reading 
 the code and discussing this with Jeffrey, we realized that the dist log 
 replay code is not compatible with rolling upgrades from 0.98.0 and 1.0.0.
 The issue is that, the region server looks at it own configuration to decide 
 whether the region should be opened in replay mode or not. The open region 
 RPC does not contain that info. So if dist log replay is enabled on master, 
 the master will assign the region and schedule replay tasks. If the region is 
 opened in a RS that does not have this conf enabled, then it will happily 
 open the region in normal mode (not replay mode) causing possible (transient) 
 data loss. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-8763) [BRAINSTORM] Combine MVCC and SeqId


[ 
https://issues.apache.org/jira/browse/HBASE-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018163#comment-14018163
 ] 

stack commented on HBASE-8763:
--

Looking at 5.1:

Why return a Pair?  We are passing in the Cell?  Can't caller use passed in 
Cell and just leave addTo returning long as before (I am missing something I 
know... pardon my being slow).  We are making a new object each time we add to 
memstore, the Pair.

Why would the below take a list of kvs to appendNoSyncNoAppend?  When would it 
ever make sense passing a list of kvs?  (Hmm... I think I see why -- when we 
want to just set an mvcc on the KV though we are not appending it -- is that 
right?  If so, a comment on the @param would help)

In the appendNoSyncNoAppend we make HLogKey.  We call System.currentTimeMillis. 
 This edit is never appended.  Can we NOT call System.currentTimeMillis?  Just 
pass a -1 or something instead?  Or can we make a noop HLogKey defined as a 
static in HLogKey and just use it every time rather than create a new one each 
time through?  Just as we have WALEdit.EMPTY_WALEDIT?

On beginMemstoreInsert, why take a value at all?  Wny not just return the 
WriteEntry that has special value for the write number?  If we ever try to use 
this number advancing the read point, throw exceptions?  Remove the current 
beginMemstoreInsert that does not take an entry?  I see that in your new 
method, waitForPreviousTransactionsComplete, you put something into the mvcc 
queue w/ a HLog.NO_SEQUENCE_ID and wait for this edit to go around so you can 
be sure queue is cleared.  So you have second use for special mvcc/sequenceid 
number.  Should the NO_SEQUENCE_ID be it?  and you just use it when 
beginMemstoreInsert is called setting it into the WriteEntry?  Should the 
number even come from HLog?  Could it be private to this class?
  
When we do waitForPreviousTransactionsComplete, does WriteEntry have a 
writeNumber set? Or is it NO_SEQUENCE_ID?  If it is the latter, yeah, just 
change beginMemstoreInsert to not take a param, or at least, not take this 
particular one because it is means of asking for a special behavior.  If the 
writeNumber is set, where does that happen?

NO_SEQUENCE_ID Should be a define in your new SequenceId interface?
 
A comment on wny you do 'w = null;' would be helpful in flush: e.g. Set to 
null to indicate success

Change name of memstoreKVs to be memstoreCells (be forward thinking!)

I am not clear still on why the below is ok up in HRegion#doMiniBatchMutation 
(Do we need the MutableLong here still? Why not just set the sequenceid into 
beginMemstoreInsertWithSeqNum and you are doing this when you use it   
kv.setMvccVersion(mvccNum.longValue());)

  mvccNum.setValue(this.sequenceId.incrementAndGet());

At a minimum it needs a comment explaining why (Sorry if I am being dense here).

You know why we add to memstore first before WAL?  For speed IIRC.  I should go 
research it.  This rollback stuff could be tricky.

So then here:

mvccNum.setValue(this.sequenceId.incrementAndGet());
w = mvcc.beginMemstoreInsertWithSeqNum(mvccNum);

We are getting a seqid and setting it as write number. We have not yet gone on 
the ring buffer.  Every edit is getting a write number like this?  MVCC read 
number happens only after the WAL append has happened.

Man, the mvcc stuff should be redone w/ disruptor.  Looks like ideal disruptor 
case.

StoreFlusher change no longer needed?

Can these be lists of Cells rather than   private final transient 
ListKeyValue memstoreKVs;?  You can do cell.setMvccVersion.

Why this?

  long appendNoSync(HTableDescriptor htd, HRegionInfo info, HLogKey key, 
WALEdit edits,
  AtomicLong sequenceId, boolean inMemstore, ListKeyValue memstoreKVs)

Unnecssary import in HLogSplitter?  Unnecessary change in WALEdit?





Are we not passing the KVs twice?  Once in WALEdits and then again in this new 
memstoreKVs argument?

I'm running tests now to see what this patch does for performance.  After our 
chat yesterday, yes, I see, it should not have much of an impact (especially 
looking at what you did in FSHLog).  That'd be cool.

I'm excited about this patch coming in.  Great work Mr. Zhong.





















 [BRAINSTORM] Combine MVCC and SeqId
 ---

 Key: HBASE-8763
 URL: https://issues.apache.org/jira/browse/HBASE-8763
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Enis Soztutar
Assignee: Jeffrey Zhong
Priority: Critical
 Attachments: HBase MVCC  LogSeqId Combined.pdf, 
 hbase-8736-poc.patch, hbase-8763-poc-v1.patch, hbase-8763-v1.patch, 
 hbase-8763-v2.patch, hbase-8763-v3.patch, hbase-8763-v4.patch, 
 hbase-8763-v5.1.patch, hbase-8763-v5.patch, hbase-8763_wip1.patch


 HBASE-8701 and a lot of recent issues include

[jira] [Commented] (HBASE-8763) [BRAINSTORM] Combine MVCC and SeqId

2014-06-04 Thread Liyin Tang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018165#comment-14018165
 ] 

Liyin Tang commented on HBASE-8763:
---

Hi, I am out of office since 9/1/2012 to 9/16/2012 and I cannot access to this 
email.
In urgent case, please forward your email to liyint...@gmail.com

Thanks a lot
Liyin


 [BRAINSTORM] Combine MVCC and SeqId
 ---

 Key: HBASE-8763
 URL: https://issues.apache.org/jira/browse/HBASE-8763
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Enis Soztutar
Assignee: Jeffrey Zhong
Priority: Critical
 Attachments: HBase MVCC  LogSeqId Combined.pdf, 
 hbase-8736-poc.patch, hbase-8763-poc-v1.patch, hbase-8763-v1.patch, 
 hbase-8763-v2.patch, hbase-8763-v3.patch, hbase-8763-v4.patch, 
 hbase-8763-v5.1.patch, hbase-8763-v5.patch, hbase-8763_wip1.patch


 HBASE-8701 and a lot of recent issues include good discussions about mvcc + 
 seqId semantics. It seems that having mvcc and the seqId complicates the 
 comparator semantics a lot in regards to flush + WAL replay + compactions + 
 delete markers and out of order puts. 
 Thinking more about it I don't think we need a MVCC write number which is 
 different than the seqId. We can keep the MVCC semantics, read point and 
 smallest read points intact, but combine mvcc write number and seqId. This 
 will allow cleaner semantics + implementation + smaller data files. 
 We can do some brainstorming for 0.98. We still have to verify that this 
 would be semantically correct, it should be so by my current understanding.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11135) Change region sequenceid generation so happens earlier in the append cycle rather than just before added to file


 [ 
https://issues.apache.org/jira/browse/HBASE-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-11135:
--

   Resolution: Fixed
Fix Version/s: 0.99.0
   Status: Resolved  (was: Patch Available)

This was committed a while back.  Resolving.

 Change region sequenceid generation so happens earlier in the append cycle 
 rather than just before added to file
 

 Key: HBASE-11135
 URL: https://issues.apache.org/jira/browse/HBASE-11135
 Project: HBase
  Issue Type: Sub-task
  Components: wal
Reporter: stack
Assignee: stack
 Fix For: 0.99.0

 Attachments: 11135.wip.txt, 11135v2.txt, 11135v5.txt, 11135v5.txt, 
 11135v5.txt, 11135v6.txt, 11135v7.txt, 11135v8.addendum.doc.txt, 11135v8.txt, 
 11135v8.txt


 Currently we assign the region edit/sequence id just before we put it in the 
 WAL.  We do it in the single thread that feeds from the ring buffer.  Doing 
 it at this point, we can ensure order, that the edits will be in the file in 
 accordance w/ the ordering of the region sequence id.
 But the point at which region sequence id is assigned an edit is deep down in 
 the WAL system and there is a lag between our putting an edit into the WAL 
 system and the edit actually getting its edit/sequence id.
 This lag -- late-binding -- complicates the unification of mvcc and region 
 sequence id, especially around async WAL writes (and, related, for no-WAL 
 writes) -- the parent for this issue (For async, how you get the edit id in 
 our system when the threads have all gone home -- unless you make them wait?)
 Chatting w/ Jeffrey Zhong yesterday, we came up with a crazypants means of 
 getting the region sequence id near-immediately.  We'll run two ringbuffers.  
 The first will mesh all handler threads and the consumer will generate ids 
 (we will have order on other side of this first ring buffer), and then if 
 async or no sync, we will just let the threads return ... updating mvcc just 
 before we let them go.  All other calls will go up on to the second ring 
 buffer to be serviced as now (batching, distribution out among the sync'ing 
 threads).  The first rb will have no friction and should turn at fast rates 
 compared to the second.  There should not be noticeable slowdown nor do I 
 foresee this refactor intefering w/ our multi-WAL plans.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-7912) HBase Backup/Restore Based on HBase Snapshot

2014-06-04 Thread Demai Ni (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Demai Ni updated HBASE-7912:

Attachment: HBaseBackupRestore-Jira-7912-DesignDoc-v2.pdf

uploaded designDoc V2 with a few minor changes, and listed limitations

HBase Backup/Restore Based on HBase Snapshot

A common practice of backup and restore in database is to first take full
baseline backup, and then periodically take incremental backup that capture
the changes since the full baseline backup. HBase cluster can store massive
amount data. Combination of full backups with incremental backups has
tremendous benefit for HBase as well. The following is a typical scenario
for full and incremental backup.
# The user takes a full backup of a table or a set of tables in HBase.
# The user schedules periodical incremental backups to capture the changes
from the full backup, or from last incremental backup.
# The user needs to restore table data to a past point of time.
# The full backup is restored to the table(s) or to different table name(s).
Then the incremental backups that are up to the desired point in time are
applied on top of the full backup.
We would support the following key features and capabilities.
* Full backup uses HBase snapshot to capture HFiles.
* Use HBase WALs to capture incremental changes, but we use bulk load of
HFiles for fast incremental restore.
* Support single table or a set of tables, and column family level backup and
restore.
* Restore to different table names.
* Support adding additional tables or CF to backup set without interruption
of incremental backup schedule.
* Support rollup/combining of incremental backups into longer period and
bigger incremental backups.
* Unified command line interface for all the above.
The solution will support HBase backup to FileSystem, either on the same
cluster or across clusters. It has the flexibility to support backup to
other devices and servers in the future.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11059) ZK-less region assignment


[ 
https://issues.apache.org/jira/browse/HBASE-11059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018262#comment-14018262
 ] 

stack commented on HBASE-11059:
---

-public class RegionState implements org.apache.hadoop.io.Writable {
+public class RegionState {

Above is good.  We serialize RegionState ever?  Nvm... I see RegionStatusProto 
now.

On:

-  ZKUtil.createAndFailSilent(this, assignmentZNode);
+  if (conf.getBoolean(hbase.assignment.usezk, false)) {
+ZKUtil.createAndFailSilent(this, assignmentZNode);
+  }


Should we remove the assignment znode dir if it exists?  (NVM... I see you do 
it on startup).

Leave of this sentence I'd say:  The other
+   * way around is not recommended also sophisticated users could do it
+   * somehow.

Put this in release note or lets open issue to ensure the below gets added to 
the 1.0 migration doc:

+   * For rolling upgrade from using ZK to not using ZK, it can be
+   * done in two steps:
+   * 1. Set hbase.assignment.usezk to true, do a rolling upgrade
+   * so that both masters and regionservers have the new code. Either
+   * master or regionserver can be upgraded at first. The order
+   * is not important for this step.
+   * 2. Set hbase.assignment.usezk to false, do a rolling restart
+   * so that region assignments don't use ZK any more. For this step,
+   * masters should be restarted after regionservers have all
+   * restarted at first so that they won't update meta table
+   * directly and master doesn't know about it.

This message will come out wrong?

+LOG.info(Finished region assignment in (failover= + failover
+  + ) + (System.currentTimeMillis() - startTime) + ms);

It will have 'failover=true' in between 'in' and time in ms... should it be on 
end?

It is a pity we have to pass this flag useZKForAssignment down:

-unassign(regionInfo, rsClosing, expectedVersion, null, true, 
null);
+unassign(regionInfo, rsClosing, expectedVersion, null, 
useZKForAssignment, null);

Can this be done in a 'transaction'?

+  mergingRegions.remove(encodedName);
+  regionOffline(a, State.MERGED);
+  regionOffline(b, State.MERGED);
+  regionOnline(p, sn, 1);

We have transactions right as long as all in the same region?  Which is the 
case when only single meta.

I skimmed the rest.

This looks amazing Jimmy.  I'd have thought it would have taken way more code.

Looking at tests, I was wondering if the new state transitions could be tested 
independent of the servers?  That possible at all?  Should be easier now no zk? 
 Can the state management classes be stood up independent of the Master?

I can see this going in if your testing works out.  We'd have it off on commit. 
 Need to add doc on new assignment state of affairs.  You have it written up in 
code.  Would just copy it into releases notes and/or into refguide.  We'd open 
new blocker against 1.0 to discuss whether to have it on for 1.0 (it'd be great 
if we could have this on... I think it'll save orders of magnitude more 
headache than it causes).  If it is on for 1.0, we'd have to doc the rolling 
upgrade.

 ZK-less region assignment
 -

 Key: HBASE-11059
 URL: https://issues.apache.org/jira/browse/HBASE-11059
 Project: HBase
  Issue Type: Improvement
  Components: master, Region Assignment
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: hbase-11059.patch, hbase-11059_v2.patch, zk-less_am.pdf


 It seems that most people don't like region assignment with ZK (HBASE-5487), 
 which causes many uncertainties. This jira is to support ZK-less region 
 assignment. We need to make sure this patch doesn't break backward 
 compatibility/rolling upgrade.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11204) Document bandwidth consumption limit feature for ExportSnapshot


[ 
https://issues.apache.org/jira/browse/HBASE-11204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018274#comment-14018274
 ] 

Hudson commented on HBASE-11204:


SUCCESS: Integrated in HBase-TRUNK #5174 (See 
[https://builds.apache.org/job/HBase-TRUNK/5174/])


 Document bandwidth consumption limit feature for ExportSnapshot
 ---

 Key: HBASE-11204
 URL: https://issues.apache.org/jira/browse/HBASE-11204
 Project: HBase
  Issue Type: Task
  Components: snapshots
Reporter: Ted Yu
Assignee: Misty Stanley-Jones
Priority: Minor
 Fix For: 0.99.0

 Attachments: HBASE-11204-1.patch, HBASE-11204-2.patch, 
 HBASE-11204.patch


 http://hbase.apache.org/book.html#ops.snapshots.export should document 
 bandwidth consumption limit feature which is implemented by HBASE-11083 and 
 HBASE-11090



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-6139) Add troubleshooting section for CentOS 6.2 page allocation failure issue


[ 
https://issues.apache.org/jira/browse/HBASE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018273#comment-14018273
 ] 

Hudson commented on HBASE-6139:
---

SUCCESS: Integrated in HBase-TRUNK #5174 (See 
[https://builds.apache.org/job/HBase-TRUNK/5174/])
HBASE-6139 Add troubleshooting section for CentOS 6.2 page allocation failure 
issue (Misty Stanley-Jones) (stack: rev 
f24e68426c9d17389258c0f501ad1f1f025971cf)
* src/main/docbkx/troubleshooting.xml


 Add troubleshooting section for CentOS 6.2 page allocation failure issue
 

 Key: HBASE-6139
 URL: https://issues.apache.org/jira/browse/HBASE-6139
 Project: HBase
  Issue Type: Task
  Components: documentation
Reporter: Andrew Purtell
Assignee: Misty Stanley-Jones
 Fix For: 0.99.0

 Attachments: HBASE-6139.patch


 Tim Robertson reports:
 bq. HBase CentOS version 6.2 reports kernel: java: page allocation failure. 
 order:4, mode:0x20. Any ideas anyone?
 Then:
 bq. echo 360448  /proc/sys/vm/min_free_kbytes appears to stop page 
 allocation failure using HBase on CentOS 6.2
 If this is the proper fix for this condition, we should document it.
 @Tim, how did you arrive at 360448?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10915) Decouple region closing (HM and HRS) from ZK


[ 
https://issues.apache.org/jira/browse/HBASE-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018275#comment-14018275
 ] 

Hudson commented on HBASE-10915:


SUCCESS: Integrated in HBase-TRUNK #5174 (See 
[https://builds.apache.org/job/HBase-TRUNK/5174/])
HBASE-10915 Decouple region closing (HM and HRS) from ZK (Mikhail Antonov) 
(stack: rev ec9c12edff8d2868a4ebb6bc76538abfb0d105d8)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/CloseMetaHandler.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZkCoordinatedStateManager.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZkCloseRegionCoordination.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/CloseRegionHandler.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/CloseRegionCoordination.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/BaseCoordinatedStateManager.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java


 Decouple region closing (HM and HRS) from ZK
 

 Key: HBASE-10915
 URL: https://issues.apache.org/jira/browse/HBASE-10915
 Project: HBase
  Issue Type: Sub-task
  Components: Consensus, Zookeeper
Affects Versions: 0.99.0
Reporter: Mikhail Antonov
Assignee: Mikhail Antonov
 Fix For: 0.99.0

 Attachments: HBASE-10915 (2).patch, HBASE-10915.patch, 
 HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, 
 HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, 
 HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, 
 HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch


 Decouple region closing from ZK. 
 Includes RS side (CloseRegionHandler), HM side (ClosedRegionHandler) and the 
 code using (HRegionServer, RSRpcServices etc).
 May need small changes in AssignmentManager.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange


[ 
https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018284#comment-14018284
 ] 

stack commented on HBASE-10771:
---

+1

Where do you confirm that the long and int writing is same as BBs?  You should 
note in the implementations that they are equivalent in a comment on commit.

I just read over this issue.  Nice back and forth.  Good stuff.

 Primitive type put/get APIs in ByteRange 
 -

 Key: HBASE-10771
 URL: https://issues.apache.org/jira/browse/HBASE-10771
 Project: HBase
  Issue Type: Improvement
Reporter: Anoop Sam John
Assignee: Anoop Sam John
 Fix For: 0.99.0

 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch, 
 HBASE-10771_V3.patch, HBASE-10771_V4.patch


 While doing HBASE-10713 I came across the need to write int/long (and read 
 also) from a ByteRange.  CellBlocks are backed by ByteRange. So we can add 
 such APIs.
 Also as per HBASE-10750  we return a ByteRange from MSLAB and also discussion 
 under HBASE-10191 suggest we can have BR backed HFileBlocks etc.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11280) Document distributed log replay and distributed log splitting


 [ 
https://issues.apache.org/jira/browse/HBASE-11280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misty Stanley-Jones updated HBASE-11280:


Attachment: HBASE-11280-1.patch

Incorporated feedback so far.

 Document distributed log replay and distributed log splitting
 -

 Key: HBASE-11280
 URL: https://issues.apache.org/jira/browse/HBASE-11280
 Project: HBase
  Issue Type: Sub-task
  Components: documentation, MTTR, wal
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
 Fix For: 0.99.0

 Attachments: HBASE-11280-1.patch, HBASE-11280.patch


 Enable 'distributed log replay' by default.  Depends on hfilev3 being enabled.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11295) Long running scan produces OutOfOrderScannerNextException

2014-06-04 Thread Anoop Sam John (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-11295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018302#comment-14018302
]

Anoop Sam John commented on HBASE-11295:

When the first request for a scan is executed in HRS the next seq id expected
(from client) is the incremented one ie. 1
Yes the out of order exp is a DNRIOE. The normal retry is from client on
client timeout.
In this case of timeout we should not allow that retry to happen. The first try
is already in progress and the scan advanced some rk levels. The retry will
skip those snd we will fail getting those rows.
On the out of order exception no retry ( like u see before) has to happen. It
will be cloding that scanner and creating a new scanner. U mean this new is
not happening? This way it was when we wrote code need to see any change made
later fir not happen.

Long running scan produces OutOfOrderScannerNextException
-

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11280) Document distributed log replay and distributed log splitting

[
https://issues.apache.org/jira/browse/HBASE-11280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018361#comment-14018361
]

Hadoop QA commented on HBASE-11280:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12648395/HBASE-11280-1.patch
against trunk revision .
ATTACHMENT ID: 12648395

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+Usually, there is only one instance of a WAL per RegionServer.
The RegionServer records Puts and Deletes to
+HBase 0.94, they were stored in
filename/hbase/.logs//filename), with subdirectories per
+ paraLog splitting is done by the HMaster during cluster start-up
or by the ServerShutdownHandler
+file. The temporary edit file is stored to disk with the
following naming pattern:/para
+ para When the region is opened, the
filenamerecovered.edits/filename folder is checked for recovered
+(link
xlink:href=https://issues.apache.org/jira/browse/HBASE-1364;HBASE-1364/link)

+[hdfs%3A%2F%2Fhost2.sample.com%3A56020%2Fhbase%2F.logs%2Fhost8.sample.com%2C57020%2C1340474893275-splitting%2Fhost8.sample.com%253A57020.1340474893900,

+hdfs%3A%2F%2Fhost2.sample.com%3A56020%2Fhbase%2F.logs%2Fhost3.sample.com%2C57020%2C1340474893299-splitting%2Fhost3.sample.com%253A57020.1340474893931,

+hdfs%3A%2F%2Fhost2.sample.com%3A56020%2Fhbase%2F.logs%2Fhost4.sample.com%2C57020%2C1340474893287-splitting%2Fhost4.sample.com%253A57020.1340474893946]

+userinputget
/hbase/splitlog/hdfs%3A%2F%2Fhost2.sample.com%3A56020%2Fhbase%2F.logs%2Fhost6.sample.com%2C57020%2C1340474893287-splitting%2Fhost6.sample.com%253A57020.1340474893945

{color:green}+1 site{color}. The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9692//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9692//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9692//console

This message is automatically generated.

Document distributed log replay and distributed log splitting
-

Attachments: HBASE-11280-1.patch, HBASE-11280.patch

Enable 'distributed log replay' by default. Depends on hfilev3 being enabled.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10641) Configurable Bucket Sizes in bucketCache


[ 
https://issues.apache.org/jira/browse/HBASE-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018364#comment-14018364
 ] 

Hudson commented on HBASE-10641:


SUCCESS: Integrated in HBase-TRUNK #5175 (See 
[https://builds.apache.org/job/HBase-TRUNK/5175/])
HBASE-10641 Configurable Bucket Sizes in bucketCache (ndimiduk: rev 
d824f0b25fbeea555257dab7e5dca7e96e8f4666)
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketCache.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java


 Configurable Bucket Sizes in bucketCache
 

 Key: HBASE-10641
 URL: https://issues.apache.org/jira/browse/HBASE-10641
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Biju Nair
Assignee: Nick Dimiduk
  Labels: cache
 Fix For: 0.99.0, 0.98.4

 Attachments: HBASE-10641.00.patch, HBASE-10641.01-0.98.patch, 
 HBASE-10641.01.patch, HBASE-10641.02-0.98.patch, HBASE-10641.02.patch


 In the current implementation of BucketCache, the sizes of buckets created 
 for different blocksizes is fixed. Given that the all the blocksizes will not 
 be used, it would be good to have the bucketCache configurable. The block 
 sizes for which buckets need to be created and the size of the buckets for 
 each block size should be configurable.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-10962) Decouple region opening (HM and HRS) from ZK


 [ 
https://issues.apache.org/jira/browse/HBASE-10962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Antonov updated HBASE-10962:


Attachment: HBASE-10962.patch

Rebased patch. updated it also on RB - https://reviews.apache.org/r/20338/

 Decouple region opening (HM and HRS) from ZK
 

 Key: HBASE-10962
 URL: https://issues.apache.org/jira/browse/HBASE-10962
 Project: HBase
  Issue Type: Sub-task
  Components: Consensus, Zookeeper
Reporter: Mikhail Antonov
Assignee: Mikhail Antonov
 Attachments: HBASE-10962.patch, HBASE-10962.patch, HBASE-10962.patch, 
 HBASE-10962.patch


 This involves creating consensus class to be kept by ConsensusProvider 
 interface, and modifications of the following classes:
  - HRS side (OpenRegionHandler, OpenMetaHandler, and few accompanying changes 
 in HRegionServer code and RsRpcServices)
  - HM side (OpenedRegionHandler, may be some changes in AssignmentManager)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-10962) Decouple region opening (HM and HRS) from ZK


 [ 
https://issues.apache.org/jira/browse/HBASE-10962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Antonov updated HBASE-10962:


Status: Patch Available  (was: Open)

 Decouple region opening (HM and HRS) from ZK
 

 Key: HBASE-10962
 URL: https://issues.apache.org/jira/browse/HBASE-10962
 Project: HBase
  Issue Type: Sub-task
  Components: Consensus, Zookeeper
Reporter: Mikhail Antonov
Assignee: Mikhail Antonov
 Attachments: HBASE-10962.patch, HBASE-10962.patch, HBASE-10962.patch, 
 HBASE-10962.patch


 This involves creating consensus class to be kept by ConsensusProvider 
 interface, and modifications of the following classes:
  - HRS side (OpenRegionHandler, OpenMetaHandler, and few accompanying changes 
 in HRegionServer code and RsRpcServices)
  - HM side (OpenedRegionHandler, may be some changes in AssignmentManager)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-10962) Decouple region opening (HM and HRS) from ZK


 [ 
https://issues.apache.org/jira/browse/HBASE-10962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Antonov updated HBASE-10962:


Status: Open  (was: Patch Available)

 Decouple region opening (HM and HRS) from ZK
 

 Key: HBASE-10962
 URL: https://issues.apache.org/jira/browse/HBASE-10962
 Project: HBase
  Issue Type: Sub-task
  Components: Consensus, Zookeeper
Reporter: Mikhail Antonov
Assignee: Mikhail Antonov
 Attachments: HBASE-10962.patch, HBASE-10962.patch, HBASE-10962.patch, 
 HBASE-10962.patch


 This involves creating consensus class to be kept by ConsensusProvider 
 interface, and modifications of the following classes:
  - HRS side (OpenRegionHandler, OpenMetaHandler, and few accompanying changes 
 in HRegionServer code and RsRpcServices)
  - HM side (OpenedRegionHandler, may be some changes in AssignmentManager)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-9733) Book should have individual Disqus comment per page


 [ 
https://issues.apache.org/jira/browse/HBASE-9733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misty Stanley-Jones updated HBASE-9733:
---

Attachment: HBASE-9733.patch

Does this work? The caveat is that I fear we will lose all the old comments. I 
wonder if there is a way we can assign the old comments to pages, or maybe just 
assign them to the book's index.html?

 Book should have individual Disqus comment per page
 ---

 Key: HBASE-9733
 URL: https://issues.apache.org/jira/browse/HBASE-9733
 Project: HBase
  Issue Type: Task
  Components: documentation
Reporter: Nick Dimiduk
Assignee: Misty Stanley-Jones
 Attachments: HBASE-9733.patch


 Instead of one blob of comments for the whole thing, each page in the book 
 should be it's own article, or whatever Disqus uses it uniquely identify an 
 comment-able entity.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (HBASE-9733) Book should have individual Disqus comment per page


 [ 
https://issues.apache.org/jira/browse/HBASE-9733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misty Stanley-Jones reassigned HBASE-9733:
--

Assignee: Misty Stanley-Jones

 Book should have individual Disqus comment per page
 ---

 Key: HBASE-9733
 URL: https://issues.apache.org/jira/browse/HBASE-9733
 Project: HBase
  Issue Type: Task
  Components: documentation
Reporter: Nick Dimiduk
Assignee: Misty Stanley-Jones
 Attachments: HBASE-9733.patch


 Instead of one blob of comments for the whole thing, each page in the book 
 should be it's own article, or whatever Disqus uses it uniquely identify an 
 comment-able entity.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-8763) [BRAINSTORM] Combine MVCC and SeqId


[ 
https://issues.apache.org/jira/browse/HBASE-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018385#comment-14018385
 ] 

stack commented on HBASE-8763:
--

This patch seems to get us our speed back.  Good on one [~jeffreyz]

Below do basic 1, 5, 25, and 200 threads test:

Here is nopatch, the current master branch:
{code}
nopatch1.1.txt:2014-06-04 12:11:52,176 INFO  [main] 
wal.HLogPerformanceEvaluation: Summary: threads=1, iterations=100, 
syncInterval=0 took 1224.313s 816.785ops/s
nopatch5.1.txt:2014-06-04 12:29:02,864 INFO  [main] 
wal.HLogPerformanceEvaluation: Summary: threads=5, iterations=100, 
syncInterval=0 took 1025.163s 4877.273ops/s
nopatch25.1.txt:2014-06-04 12:53:02,973 INFO  [main] 
wal.HLogPerformanceEvaluation: Summary: threads=25, iterations=100, 
syncInterval=0 took 1434.641s 17425.963ops/s
nopatch200.1.txt:2014-06-04 13:40:30,333 INFO  [main] 
wal.HLogPerformanceEvaluation: Summary: threads=200, iterations=100, 
syncInterval=0 took 2841.947s 70374.289ops/s
{code}

Here is w/ patch applied:
{code}
patch1.1.txt:2014-06-04 14:37:04,973 INFO  [main] 
wal.HLogPerformanceEvaluation: Summary: threads=1, iterations=100, 
syncInterval=0 took 1228.775s 813.819ops/s
patch5.1.txt:2014-06-04 14:53:53,623 INFO  [main] 
wal.HLogPerformanceEvaluation: Summary: threads=5, iterations=100, 
syncInterval=0 took 1003.234s 4983.882ops/s
patch25.1.txt:2014-06-04 15:17:17,952 INFO  [main] 
wal.HLogPerformanceEvaluation: Summary: threads=25, iterations=100, 
syncInterval=0 took 1398.927s 17870.840ops/s
patch200.1.txt:2014-06-04 15:47:36,297 INFO  [main] 
wal.HLogPerformanceEvaluation: Summary: threads=200, iterations=100, 
syncInterval=0 took 1813.013s 110313.609ops/s
{code}

 [BRAINSTORM] Combine MVCC and SeqId
 ---

 Key: HBASE-8763
 URL: https://issues.apache.org/jira/browse/HBASE-8763
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Enis Soztutar
Assignee: Jeffrey Zhong
Priority: Critical
 Attachments: HBase MVCC  LogSeqId Combined.pdf, 
 hbase-8736-poc.patch, hbase-8763-poc-v1.patch, hbase-8763-v1.patch, 
 hbase-8763-v2.patch, hbase-8763-v3.patch, hbase-8763-v4.patch, 
 hbase-8763-v5.1.patch, hbase-8763-v5.patch, hbase-8763_wip1.patch


 HBASE-8701 and a lot of recent issues include good discussions about mvcc + 
 seqId semantics. It seems that having mvcc and the seqId complicates the 
 comparator semantics a lot in regards to flush + WAL replay + compactions + 
 delete markers and out of order puts. 
 Thinking more about it I don't think we need a MVCC write number which is 
 different than the seqId. We can keep the MVCC semantics, read point and 
 smallest read points intact, but combine mvcc write number and seqId. This 
 will allow cleaner semantics + implementation + smaller data files. 
 We can do some brainstorming for 0.98. We still have to verify that this 
 would be semantically correct, it should be so by my current understanding.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-9733) Book should have individual Disqus comment per page


[ 
https://issues.apache.org/jira/browse/HBASE-9733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018386#comment-14018386
 ] 

stack commented on HBASE-9733:
--

I think as long as old comments addressed, it is fine to let them go (I think 
they are answered...)

 Book should have individual Disqus comment per page
 ---

 Key: HBASE-9733
 URL: https://issues.apache.org/jira/browse/HBASE-9733
 Project: HBase
  Issue Type: Task
  Components: documentation
Reporter: Nick Dimiduk
Assignee: Misty Stanley-Jones
 Attachments: HBASE-9733.patch


 Instead of one blob of comments for the whole thing, each page in the book 
 should be it's own article, or whatever Disqus uses it uniquely identify an 
 comment-able entity.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HBASE-9733) Book should have individual Disqus comment per page


 [ 
https://issues.apache.org/jira/browse/HBASE-9733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-9733.
--

   Resolution: Fixed
Fix Version/s: 0.99.0
 Hadoop Flags: Reviewed

This worked for me testing locally.  Committing.

 Book should have individual Disqus comment per page
 ---

 Key: HBASE-9733
 URL: https://issues.apache.org/jira/browse/HBASE-9733
 Project: HBase
  Issue Type: Task
  Components: documentation
Reporter: Nick Dimiduk
Assignee: Misty Stanley-Jones
 Fix For: 0.99.0

 Attachments: HBASE-9733.patch


 Instead of one blob of comments for the whole thing, each page in the book 
 should be it's own article, or whatever Disqus uses it uniquely identify an 
 comment-able entity.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-7050) Document locking of HRegion


[ 
https://issues.apache.org/jira/browse/HBASE-7050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018403#comment-14018403
 ] 

Misty Stanley-Jones commented on HBASE-7050:


Can you share what you have come up with so far? Or a good source for this info?

 Document locking of HRegion
 ---

 Key: HBASE-7050
 URL: https://issues.apache.org/jira/browse/HBASE-7050
 Project: HBase
  Issue Type: Task
  Components: documentation, Transactions/MVCC
Reporter: Gregory Chanan
Priority: Minor

 Would be good to have some documentation on the locking in HRegion.  In 
 particular, the updateLock and the row locks.
 With the optimizations that have been made, e.g. HBASE-5541 this isn't 
 straightforward.
 When do these locks need to be held?  What structures do they protect access 
 to?  Why does the updateLock not need to be held during rollback?  Those 
 sorts of questions.
 I've started do to some of this, but haven't fully fleshed it out.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-7050) Document locking of HRegion

2014-06-04 Thread Gregory Chanan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018415#comment-14018415
 ] 

Gregory Chanan commented on HBASE-7050:
---

Hmm...unfortunately I can't find my notes anymore.  And they would probably be 
too out of date to be reliable.

At the time, I found the following the most useful:
1)  The HBase Internals talk by Lars 
(http://www.slideshare.net/cloudera/3-learning-h-base-internals-lars-hofhansl-salesforce-final)
2) Reading through HRegion.java and trying to understand everything

There may be better resources now, I'm not sure.  Hopefully that helps. 

 Document locking of HRegion
 ---

 Key: HBASE-7050
 URL: https://issues.apache.org/jira/browse/HBASE-7050
 Project: HBase
  Issue Type: Task
  Components: documentation, Transactions/MVCC
Reporter: Gregory Chanan
Priority: Minor

 Would be good to have some documentation on the locking in HRegion.  In 
 particular, the updateLock and the row locks.
 With the optimizations that have been made, e.g. HBASE-5541 this isn't 
 straightforward.
 When do these locks need to be held?  What structures do they protect access 
 to?  Why does the updateLock not need to be held during rollback?  Those 
 sorts of questions.
 I've started do to some of this, but haven't fully fleshed it out.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HBASE-11147) Avoid creating List of KeyValue in FilterBase#filterRowCells(ListCell)

2014-06-04 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-11147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HBASE-11147.


Resolution: Later

 Avoid creating List of KeyValue in FilterBase#filterRowCells(ListCell)
 

 Key: HBASE-11147
 URL: https://issues.apache.org/jira/browse/HBASE-11147
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Gustavo Anatoly
Priority: Trivial

 Currently a List of KeyValue's is always created:
 {code}
 ListKeyValue kvs = new ArrayListKeyValue(ignored.size());
 {code}
 When passed ignored List is of KeyValue (which is the only implementation of 
 Cell at the moment), the above step should be avoided.
 This would reduce creation of short-lived objects.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10641) Configurable Bucket Sizes in bucketCache


[ 
https://issues.apache.org/jira/browse/HBASE-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018426#comment-14018426
 ] 

Hudson commented on HBASE-10641:


SUCCESS: Integrated in HBase-0.98 #328 (See 
[https://builds.apache.org/job/HBase-0.98/328/])
HBASE-10641 Configurable Bucket Sizes in bucketCache (ndimiduk: rev 
2c800bd68fbcc02dfef6ecdd09bb50f3c898231a)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketCache.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java


 Configurable Bucket Sizes in bucketCache
 

 Key: HBASE-10641
 URL: https://issues.apache.org/jira/browse/HBASE-10641
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Biju Nair
Assignee: Nick Dimiduk
  Labels: cache
 Fix For: 0.99.0, 0.98.4

 Attachments: HBASE-10641.00.patch, HBASE-10641.01-0.98.patch, 
 HBASE-10641.01.patch, HBASE-10641.02-0.98.patch, HBASE-10641.02.patch


 In the current implementation of BucketCache, the sizes of buckets created 
 for different blocksizes is fixed. Given that the all the blocksizes will not 
 be used, it would be good to have the bucketCache configurable. The block 
 sizes for which buckets need to be created and the size of the buckets for 
 each block size should be configurable.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10962) Decouple region opening (HM and HRS) from ZK


[ 
https://issues.apache.org/jira/browse/HBASE-10962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018427#comment-14018427
 ] 

Hadoop QA commented on HBASE-10962:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12648407/HBASE-10962.patch
  against trunk revision .
  ATTACHMENT ID: 12648407

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 15 new 
or modified tests.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 

 {color:red}-1 core zombie tests{color}.  There are 3 zombie test(s):   
at 
org.apache.hadoop.hbase.master.TestAssignmentManager.testBalance(TestAssignmentManager.java:423)
at 
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testCompactionRecordDoesntBlockRolling(TestLogRolling.java:618)
at 
org.apache.hadoop.hbase.TestDrainingServer.testAssignmentManagerDoesntUseDrainedServerWithBulkAssign(TestDrainingServer.java:264)

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9693//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9693//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9693//console

This message is automatically generated.

 Decouple region opening (HM and HRS) from ZK
 

 Key: HBASE-10962
 URL: https://issues.apache.org/jira/browse/HBASE-10962
 Project: HBase
  Issue Type: Sub-task
  Components: Consensus, Zookeeper
Reporter: Mikhail Antonov
Assignee: Mikhail Antonov
 Attachments: HBASE-10962.patch, HBASE-10962.patch, HBASE-10962.patch, 
 HBASE-10962.patch


 This involves creating consensus class to be kept by ConsensusProvider 
 interface, and modifications of the following classes:
  - HRS side (OpenRegionHandler, OpenMetaHandler, and few accompanying changes 
 in HRegionServer code and RsRpcServices)
  - HM side (OpenedRegionHandler, may be some changes in AssignmentManager)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-10289) Avoid random port usage by default JMX Server. Create Custome JMX server

2014-06-04 Thread Qiang Tian (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-10289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qiang Tian updated HBASE-10289:
---

Status: Patch Available  (was: Reopened)

 Avoid random port usage by default JMX Server. Create Custome JMX server
 

 Key: HBASE-10289
 URL: https://issues.apache.org/jira/browse/HBASE-10289
 Project: HBase
  Issue Type: Improvement
Reporter: nijel
Assignee: Qiang Tian
Priority: Minor
  Labels: stack
 Fix For: 0.99.0

 Attachments: HBASE-10289-v4.patch, HBASE-10289.patch, 
 HBASE-10289_1.patch, HBASE-10289_2.patch, HBASE-10289_3.patch, 
 HBase10289-master.patch, hbase10289-0.98.patch, 
 hbase10289-doc_update-master.patch, hbase10289-master-v1.patch, 
 hbase10289-master-v2.patch


 If we enable JMX MBean server for HMaster or Region server  through VM 
 arguments, the process will use one random which we cannot configure.
 This can be a problem if that random port is configured for some other 
 service.
 This issue can be avoided by supporting  a custom JMX Server.
 The ports can be configured. If there is no ports configured, it will 
 continue the same way as now.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-10289) Avoid random port usage by default JMX Server. Create Custome JMX server

2014-06-04 Thread Qiang Tian (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-10289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qiang Tian updated HBASE-10289:
---

Attachment: hbase10289-0.98.patch
hbase10289-doc_update-master.patch

upload the doc update for master and code patch for 0.98. (sorry for late 
response, got some emergency yesterday)

 Avoid random port usage by default JMX Server. Create Custome JMX server
 

 Key: HBASE-10289
 URL: https://issues.apache.org/jira/browse/HBASE-10289
 Project: HBase
  Issue Type: Improvement
Reporter: nijel
Assignee: Qiang Tian
Priority: Minor
  Labels: stack
 Fix For: 0.99.0

 Attachments: HBASE-10289-v4.patch, HBASE-10289.patch, 
 HBASE-10289_1.patch, HBASE-10289_2.patch, HBASE-10289_3.patch, 
 HBase10289-master.patch, hbase10289-0.98.patch, 
 hbase10289-doc_update-master.patch, hbase10289-master-v1.patch, 
 hbase10289-master-v2.patch


 If we enable JMX MBean server for HMaster or Region server  through VM 
 arguments, the process will use one random which we cannot configure.
 This can be a problem if that random port is configured for some other 
 service.
 This issue can be avoided by supporting  a custom JMX Server.
 The ports can be configured. If there is no ports configured, it will 
 continue the same way as now.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10289) Avoid random port usage by default JMX Server. Create Custome JMX server

[
https://issues.apache.org/jira/browse/HBASE-10289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018433#comment-14018433
]

Hadoop QA commented on HBASE-10289:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12648419/hbase10289-0.98.patch
against trunk revision .
ATTACHMENT ID: 12648419

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.

{color:red}-1 patch{color}. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9694//console

This message is automatically generated.

Avoid random port usage by default JMX Server. Create Custome JMX server

Key: HBASE-10289
URL: https://issues.apache.org/jira/browse/HBASE-10289
Project: HBase
Issue Type: Improvement
Reporter: nijel
Assignee: Qiang Tian
Priority: Minor
Labels: stack
Fix For: 0.99.0

Attachments: HBASE-10289-v4.patch, HBASE-10289.patch,
HBASE-10289_1.patch, HBASE-10289_2.patch, HBASE-10289_3.patch,
HBase10289-master.patch, hbase10289-0.98.patch,
hbase10289-doc_update-master.patch, hbase10289-master-v1.patch,
hbase10289-master-v2.patch

If we enable JMX MBean server for HMaster or Region server through VM
arguments, the process will use one random which we cannot configure.
This can be a problem if that random port is configured for some other
service.
This issue can be avoided by supporting a custom JMX Server.
The ports can be configured. If there is no ports configured, it will
continue the same way as now.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10641) Configurable Bucket Sizes in bucketCache


[ 
https://issues.apache.org/jira/browse/HBASE-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018445#comment-14018445
 ] 

Hudson commented on HBASE-10641:


FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #310 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/310/])
HBASE-10641 Configurable Bucket Sizes in bucketCache (ndimiduk: rev 
2c800bd68fbcc02dfef6ecdd09bb50f3c898231a)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketCache.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java


 Configurable Bucket Sizes in bucketCache
 

 Key: HBASE-10641
 URL: https://issues.apache.org/jira/browse/HBASE-10641
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Biju Nair
Assignee: Nick Dimiduk
  Labels: cache
 Fix For: 0.99.0, 0.98.4

 Attachments: HBASE-10641.00.patch, HBASE-10641.01-0.98.patch, 
 HBASE-10641.01.patch, HBASE-10641.02-0.98.patch, HBASE-10641.02.patch


 In the current implementation of BucketCache, the sizes of buckets created 
 for different blocksizes is fixed. Given that the all the blocksizes will not 
 be used, it would be good to have the bucketCache configurable. The block 
 sizes for which buckets need to be created and the size of the buckets for 
 each block size should be configurable.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-9733) Book should have individual Disqus comment per page


[ 
https://issues.apache.org/jira/browse/HBASE-9733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018446#comment-14018446
 ] 

Hudson commented on HBASE-9733:
---

SUCCESS: Integrated in HBase-TRUNK #5176 (See 
[https://builds.apache.org/job/HBase-TRUNK/5176/])
HBASE-9733 Book should have individual Disqus comment per page (Misty 
Stanley-Jones) (stack: rev 475b1d2c943ce3f47d8a3595db46d55bbf0d2abf)
* src/main/docbkx/customization.xsl


 Book should have individual Disqus comment per page
 ---

 Key: HBASE-9733
 URL: https://issues.apache.org/jira/browse/HBASE-9733
 Project: HBase
  Issue Type: Task
  Components: documentation
Reporter: Nick Dimiduk
Assignee: Misty Stanley-Jones
 Fix For: 0.99.0

 Attachments: HBASE-9733.patch


 Instead of one blob of comments for the whole thing, each page in the book 
 should be it's own article, or whatever Disqus uses it uniquely identify an 
 comment-able entity.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10861) Supporting API in ByteRange


[ 
https://issues.apache.org/jira/browse/HBASE-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018453#comment-14018453
 ] 

ramkrishna.s.vasudevan commented on HBASE-10861:


An reviews here?

 Supporting API in ByteRange
 ---

 Key: HBASE-10861
 URL: https://issues.apache.org/jira/browse/HBASE-10861
 Project: HBase
  Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Attachments: HBASE-10861.patch, HBASE-10861_2.patch, 
 HBASE-10861_3.patch


 We would need APIs that would 
 setLimit(int limit)
 getLimt()
 asReadOnly()
 These APIs would help in implementations that have Buffers offheap (for now 
 BRs backed by DBB).
 If anything more is needed could be added when needed.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-10885) Support visibility expressions on Deletes


 [ 
https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-10885:
---

Status: Patch Available  (was: Open)

 Support visibility expressions on Deletes
 -

 Key: HBASE-10885
 URL: https://issues.apache.org/jira/browse/HBASE-10885
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.98.1
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
 Fix For: 0.99.0, 0.98.4

 Attachments: HBASE-10885_1.patch, HBASE-10885_2.patch, 
 HBASE-10885_new_tag_type_1.patch, HBASE-10885_new_tag_type_2.patch, 
 HBASE-10885_v1.patch, HBASE-10885_v2.patch


 Accumulo can specify visibility expressions for delete markers. During 
 compaction the cells covered by the tombstone are determined in part by 
 matching the visibility expression. This is useful for the use case of data 
 set coalescing, where entries from multiple data sets carrying different 
 labels are combined into one common large table. Later, a subset of entries 
 can be conveniently removed using visibility expressions.
 Currently doing the same in HBase would only be possible with a custom 
 coprocessor. Otherwise, a Delete will affect all cells covered by the 
 tombstone regardless of any visibility expression scoping. This is correct 
 behavior in that no data spill is possible, but certainly could be 
 surprising, and is only meant to be transitional. We decided not to support 
 visibility expressions on Deletes to control the complexity of the initial 
 implementation.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-10885) Support visibility expressions on Deletes

[
https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

ramkrishna.s.vasudevan updated HBASE-10885:
---

Attachment: HBASE-10885_v2.patch

Updated patch after HBASE-11126 went in. Tested the patch with an IT case that
am working on internally. Will also add the IT patch once it is tested fully.
Will add it to the RB also.

Support visibility expressions on Deletes
-

Key: HBASE-10885
URL: https://issues.apache.org/jira/browse/HBASE-10885
Project: HBase
Issue Type: Improvement
Affects Versions: 0.98.1
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.99.0, 0.98.4

Attachments: HBASE-10885_1.patch, HBASE-10885_2.patch,
HBASE-10885_new_tag_type_1.patch, HBASE-10885_new_tag_type_2.patch,
HBASE-10885_v1.patch, HBASE-10885_v2.patch

Accumulo can specify visibility expressions for delete markers. During
compaction the cells covered by the tombstone are determined in part by
matching the visibility expression. This is useful for the use case of data
set coalescing, where entries from multiple data sets carrying different
labels are combined into one common large table. Later, a subset of entries
can be conveniently removed using visibility expressions.
Currently doing the same in HBase would only be possible with a custom
coprocessor. Otherwise, a Delete will affect all cells covered by the
tombstone regardless of any visibility expression scoping. This is correct
behavior in that no data spill is possible, but certainly could be
surprising, and is only meant to be transitional. We decided not to support
visibility expressions on Deletes to control the complexity of the initial
implementation.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-10885) Support visibility expressions on Deletes


 [ 
https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-10885:
---

Status: Open  (was: Patch Available)

 Support visibility expressions on Deletes
 -

 Key: HBASE-10885
 URL: https://issues.apache.org/jira/browse/HBASE-10885
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.98.1
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
 Fix For: 0.99.0, 0.98.4

 Attachments: HBASE-10885_1.patch, HBASE-10885_2.patch, 
 HBASE-10885_new_tag_type_1.patch, HBASE-10885_new_tag_type_2.patch, 
 HBASE-10885_v1.patch, HBASE-10885_v2.patch


 Accumulo can specify visibility expressions for delete markers. During 
 compaction the cells covered by the tombstone are determined in part by 
 matching the visibility expression. This is useful for the use case of data 
 set coalescing, where entries from multiple data sets carrying different 
 labels are combined into one common large table. Later, a subset of entries 
 can be conveniently removed using visibility expressions.
 Currently doing the same in HBase would only be possible with a custom 
 coprocessor. Otherwise, a Delete will affect all cells covered by the 
 tombstone regardless of any visibility expression scoping. This is correct 
 behavior in that no data spill is possible, but certainly could be 
 surprising, and is only meant to be transitional. We decided not to support 
 visibility expressions on Deletes to control the complexity of the initial 
 implementation.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10861) Supporting API in ByteRange


[ 
https://issues.apache.org/jira/browse/HBASE-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018458#comment-14018458
 ] 

Hadoop QA commented on HBASE-10861:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12644649/HBASE-10861_3.patch
  against trunk revision .
  ATTACHMENT ID: 12644649

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 56 new 
or modified tests.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9695//console

This message is automatically generated.

 Supporting API in ByteRange
 ---

 Key: HBASE-10861
 URL: https://issues.apache.org/jira/browse/HBASE-10861
 Project: HBase
  Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Attachments: HBASE-10861.patch, HBASE-10861_2.patch, 
 HBASE-10861_3.patch


 We would need APIs that would 
 setLimit(int limit)
 getLimt()
 asReadOnly()
 These APIs would help in implementations that have Buffers offheap (for now 
 BRs backed by DBB).
 If anything more is needed could be added when needed.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10933) hbck -fixHdfsOrphans is not working properly it throws null pointer exception

2014-06-04 Thread Kashif J S (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-10933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018472#comment-14018472
 ] 

Kashif J S commented on HBASE-10933:


Kindly review the patch

 hbck -fixHdfsOrphans is not working properly it throws null pointer exception
 -

 Key: HBASE-10933
 URL: https://issues.apache.org/jira/browse/HBASE-10933
 Project: HBase
  Issue Type: Bug
  Components: hbck
Affects Versions: 0.94.16, 0.98.2
Reporter: Deepak Sharma
Assignee: Kashif J S
Priority: Critical
 Fix For: 0.99.0, 0.94.21

 Attachments: HBASE-10933-0.94-v1.patch, HBASE-10933-0.94-v2.patch, 
 HBASE-10933-trunk-v1.patch, TestResults-0.94.txt, TestResults-trunk.txt


 if we regioninfo file is not existing in hbase region then if we run hbck 
 repair or hbck -fixHdfsOrphans
 then it is not able to resolve this problem it throws null pointer exception
 {code}
 2014-04-08 20:11:49,750 INFO  [main] util.HBaseFsck 
 (HBaseFsck.java:adoptHdfsOrphans(470)) - Attempting to handle orphan hdfs 
 dir: 
 hdfs://10.18.40.28:54310/hbase/TestHdfsOrphans1/5a3de9ca65e587cb05c9384a3981c950
 java.lang.NullPointerException
   at 
 org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$000(HBaseFsck.java:1939)
   at 
 org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:497)
   at 
 org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:471)
   at 
 org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:591)
   at 
 org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:369)
   at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:447)
   at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:3769)
   at org.apache.hadoop.hbase.util.HBaseFsck.run(HBaseFsck.java:3587)
   at 
 com.huawei.isap.test.smartump.hadoop.hbase.HbaseHbckRepair.repairToFixHdfsOrphans(HbaseHbckRepair.java:244)
   at 
 com.huawei.isap.test.smartump.hadoop.hbase.HbaseHbckRepair.setUp(HbaseHbckRepair.java:84)
   at junit.framework.TestCase.runBare(TestCase.java:132)
   at junit.framework.TestResult$1.protect(TestResult.java:110)
   at junit.framework.TestResult.runProtected(TestResult.java:128)
   at junit.framework.TestResult.run(TestResult.java:113)
   at junit.framework.TestCase.run(TestCase.java:124)
   at junit.framework.TestSuite.runTest(TestSuite.java:243)
   at junit.framework.TestSuite.run(TestSuite.java:238)
   at 
 org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
   at 
 org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
   at 
 org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
 {code}
 problem i got it is because since in HbaseFsck class 
 {code}
  private void adoptHdfsOrphan(HbckInfo hi)
 {code}
 we are intializing tableinfo using SortedMapString, TableInfo tablesInfo 
 object
 {code}
 TableInfo tableInfo = tablesInfo.get(tableName);
 {code}
 but  in private SortedMapString, TableInfo loadHdfsRegionInfos()
 {code}
  for (HbckInfo hbi: hbckInfos) {
   if (hbi.getHdfsHRI() == null) {
 // was an orphan
 continue;
   }
 {code}
 we have check if a region is orphan then that table will can not be added in 
 SortedMapString, TableInfo tablesInfo
 so later while using this we get null pointer exception



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10933) hbck -fixHdfsOrphans is not working properly it throws null pointer exception


[ 
https://issues.apache.org/jira/browse/HBASE-10933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018475#comment-14018475
 ] 

Hadoop QA commented on HBASE-10933:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12646009/HBASE-10933-0.94-v2.patch
  against trunk revision .
  ATTACHMENT ID: 12646009

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 9 new 
or modified tests.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9697//console

This message is automatically generated.

 hbck -fixHdfsOrphans is not working properly it throws null pointer exception
 -

 Key: HBASE-10933
 URL: https://issues.apache.org/jira/browse/HBASE-10933
 Project: HBase
  Issue Type: Bug
  Components: hbck
Affects Versions: 0.94.16, 0.98.2
Reporter: Deepak Sharma
Assignee: Kashif J S
Priority: Critical
 Fix For: 0.99.0, 0.94.21

 Attachments: HBASE-10933-0.94-v1.patch, HBASE-10933-0.94-v2.patch, 
 HBASE-10933-trunk-v1.patch, TestResults-0.94.txt, TestResults-trunk.txt


 if we regioninfo file is not existing in hbase region then if we run hbck 
 repair or hbck -fixHdfsOrphans
 then it is not able to resolve this problem it throws null pointer exception
 {code}
 2014-04-08 20:11:49,750 INFO  [main] util.HBaseFsck 
 (HBaseFsck.java:adoptHdfsOrphans(470)) - Attempting to handle orphan hdfs 
 dir: 
 hdfs://10.18.40.28:54310/hbase/TestHdfsOrphans1/5a3de9ca65e587cb05c9384a3981c950
 java.lang.NullPointerException
   at 
 org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$000(HBaseFsck.java:1939)
   at 
 org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:497)
   at 
 org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:471)
   at 
 org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:591)
   at 
 org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:369)
   at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:447)
   at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:3769)
   at org.apache.hadoop.hbase.util.HBaseFsck.run(HBaseFsck.java:3587)
   at 
 com.huawei.isap.test.smartump.hadoop.hbase.HbaseHbckRepair.repairToFixHdfsOrphans(HbaseHbckRepair.java:244)
   at 
 com.huawei.isap.test.smartump.hadoop.hbase.HbaseHbckRepair.setUp(HbaseHbckRepair.java:84)
   at junit.framework.TestCase.runBare(TestCase.java:132)
   at junit.framework.TestResult$1.protect(TestResult.java:110)
   at junit.framework.TestResult.runProtected(TestResult.java:128)
   at junit.framework.TestResult.run(TestResult.java:113)
   at junit.framework.TestCase.run(TestCase.java:124)
   at junit.framework.TestSuite.runTest(TestSuite.java:243)
   at junit.framework.TestSuite.run(TestSuite.java:238)
   at 
 org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
   at 
 org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
   at 
 org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
 {code}
 problem i got it is because since in HbaseFsck class 
 {code}
  private void adoptHdfsOrphan(HbckInfo hi)
 {code}
 we are intializing tableinfo using SortedMapString, TableInfo tablesInfo 
 object
 {code}
 TableInfo tableInfo = tablesInfo.get(tableName);
 {code}
 but  in private SortedMapString, TableInfo loadHdfsRegionInfos()
 {code}
  for (HbckInfo hbi: hbckInfos) {
   if (hbi.getHdfsHRI() == null) {
 // was an orphan
 continue;
   }
 {code}
 we have check if a region is orphan then that table will can not be added in 
 SortedMapString, TableInfo tablesInfo
 so later while using this we get null pointer exception



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10885) Support visibility expressions on Deletes