date:20120510

[
https://issues.apache.org/jira/browse/HBASE-5916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272673#comment-13272673
]

Hadoop QA commented on HBASE-5916:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12526389/HBASE-5916_trunk_4.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.TestDrainingServer

org.apache.hadoop.hbase.regionserver.TestRSKilledWhenMasterInitializing

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1836//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1836//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1836//console

This message is automatically generated.

RS restart just before master intialization we make the cluster non operative
-

Key: HBASE-5916
URL: https://issues.apache.org/jira/browse/HBASE-5916
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.94.1

Attachments: HBASE-5916_trunk.patch, HBASE-5916_trunk_1.patch,
HBASE-5916_trunk_1.patch, HBASE-5916_trunk_2.patch, HBASE-5916_trunk_3.patch,
HBASE-5916_trunk_4.patch

Consider a case where my master is getting restarted. RS that was alive when
the master restart started, gets restarted before the master initializes the
ServerShutDownHandler.
{code}
serverShutdownHandlerEnabled = true;
{code}
In this case when the RS tries to register with the master, the master will
try to expire the server but the server cannot be expired as still the
serverShutdownHandler is not enabled.
This case may happen when i have only one RS gets restarted or all the RS
gets restarted at the same time.(before assignRootandMeta).
{code}
LOG.info(message);
if (existingServer.getStartcode() serverName.getStartcode()) {
LOG.info(Triggering server recovery; existingServer +
existingServer + looks stale, new server: + serverName);
expireServer(existingServer);
}
{code}
If another RS is brought up then the cluster comes back to normalcy.
May be a very corner case.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5975) Failed suppression of fs shutdown hook with Hadoop 2.0.0


[ 
https://issues.apache.org/jira/browse/HBASE-5975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272708#comment-13272708
 ] 

Zhihong Yu commented on HBASE-5975:
---

Integrated to 0.94 and trunk.

Thanks for the patch Jimmy.

Thanks for the review, Andy.

 Failed suppression of fs shutdown hook with Hadoop 2.0.0
 

 Key: HBASE-5975
 URL: https://issues.apache.org/jira/browse/HBASE-5975
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0, 0.94.1

 Attachments: 5975+5966.patch, hbase-5975.patch


 Unit test failed with error:  Failed suppression of fs shutdown hook
 ShutdownHookManager.deleteShutdownHook failed to delete the hook since it 
 should be runnable instead of a thread for HADOOP 2.0.0.
 For other HADOOP version, runnable should work too since thread implements 
 runnable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5045) Add the table name and cf name for the next call int the task monitor

[
https://issues.apache.org/jira/browse/HBASE-5045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272712#comment-13272712
]

Hadoop QA commented on HBASE-5045:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12526400/D3045.3.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 2 new or modified tests.

-1 patch. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1838//console

This message is automatically generated.

Add the table name and cf name for the next call int the task monitor
-

Key: HBASE-5045
URL: https://issues.apache.org/jira/browse/HBASE-5045
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Amir Shimoni
Attachments: D2913.1.patch, D2913.2.patch, D3045.1.patch,
D3045.2.patch, D3045.3.patch

In the task monitor, we don't have much information about the next call
compared to other operations.
It would be nice to add the table name and cf name for each next call in the
task monitor.

[jira] [Updated] (HBASE-5955) Guava 11 drops MapEvictionListener and Hadoop 2.0.0-alpha requires it


 [ 
https://issues.apache.org/jira/browse/HBASE-5955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5955:
-

Fix Version/s: 0.94.1

 Guava 11 drops MapEvictionListener and Hadoop 2.0.0-alpha requires it
 -

 Key: HBASE-5955
 URL: https://issues.apache.org/jira/browse/HBASE-5955
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0
Reporter: Andrew Purtell
Assignee: Lars Hofhansl
 Fix For: 0.94.1


 Hadoop 2.0.0-alpha depends on Guava 11.0.2. Updating HBase dependencies to 
 match produces the following compilation errors:
 {code}
 [ERROR] SingleSizeCache.java:[41,32] cannot find symbol
 [ERROR] symbol  : class MapEvictionListener
 [ERROR] location: package com.google.common.collect
 [ERROR] 
 [ERROR] SingleSizeCache.java:[94,4] cannot find symbol
 [ERROR] symbol  : class MapEvictionListener
 [ERROR] location: class org.apache.hadoop.hbase.io.hfile.slab.SingleSizeCache
 [ERROR] 
 [ERROR] SingleSizeCache.java:[94,69] cannot find symbol
 [ERROR] symbol  : class MapEvictionListener
 [ERROR] location: class org.apache.hadoop.hbase.io.hfile.slab.SingleSizeCache
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5985) TestMetaMigrationRemovingHTD failed with HADOOP 2.0.0


[ 
https://issues.apache.org/jira/browse/HBASE-5985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272711#comment-13272711
 ] 

Hadoop QA commented on HBASE-5985:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12526392/hbase-5985.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.coprocessor.TestClassLoading
  org.apache.hadoop.hbase.TestDrainingServer

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1837//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1837//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1837//console

This message is automatically generated.

 TestMetaMigrationRemovingHTD failed with HADOOP 2.0.0
 -

 Key: HBASE-5985
 URL: https://issues.apache.org/jira/browse/HBASE-5985
 Project: HBase
  Issue Type: Test
  Components: test
Affects Versions: 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: hbase-5985.patch


 ---
 Test set: org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD
 ---
 Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.448 sec  
 FAILURE!
 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD  Time elapsed: 0 
 sec   ERROR!
 java.io.IOException: Failed put; errcode=1
 at 
 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD.doFsCommand(TestMetaMigrationRemovingHTD.java:124)
 at 
 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD.setUpBeforeClass(TestMetaMigrationRemovingHTD.java:80)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at 
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
 at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
 at 
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
 at 
 org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
 at 
 org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
 at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
 at 
 org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
 at 
 org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5898) Consider double-checked locking for block cache lock


[ 
https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272716#comment-13272716
 ] 

Todd Lipcon commented on HBASE-5898:


Yep, I agree we need to fix the metrics messup before this is a candidate for 
commit.

I like the idea of moving the metric updates out.

 Consider double-checked locking for block cache lock
 

 Key: HBASE-5898
 URL: https://issues.apache.org/jira/browse/HBASE-5898
 Project: HBase
  Issue Type: Improvement
  Components: performance
Affects Versions: 0.94.1
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Critical
 Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt


 Running a workload with a high query rate against a dataset that fits in 
 cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by 
 HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a 
 lot of CPU doing lock management here. I wrote a quick patch to switch to a 
 double-checked locking and it improved throughput substantially for this 
 workload.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB


[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272718#comment-13272718
 ] 

Todd Lipcon commented on HBASE-5453:


Agree it would be great if HFile used protobuf where possible. But I don't 
think it needs to block 0.96, since we already have decent compatibility paths 
here, and the writables used are already fairly generic (mostly maps if I 
remember correctly).

 Switch on-disk formats (reference files, HFile meta fields, etc) to PB
 --

 Key: HBASE-5453
 URL: https://issues.apache.org/jira/browse/HBASE-5453
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, master, migration, regionserver
Reporter: Todd Lipcon
Assignee: stack
 Attachments: 5453.txt, 5453v2.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

2012-05-10 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272727#comment-13272727
 ] 

jirapos...@reviews.apache.org commented on HBASE-5547:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4633/
---

(Updated 2012-05-10 20:28:50.691971)


Review request for hbase, Michael Stack and Lars Hofhansl.


Changes
---

Refactoring as per stack's comments. Also, updated the testing to avoid 
flapping. 

I think this is pretty close to ready for commit.


Summary
---

Essentially, whenever an hfile would be deleted, it is instead moved to the 
archive directory. In this impl, the archive directory is on a per table basis, 
but defaults to '.archive'. Removing hfiles occurs in three places - 
compaction, merge and catalog janitor. The former and two latter are distinctly 
different code paths, but but did pull out some similarities. The latter two 
end up calling the same method, so there should be a reasonable amount of 
overlap.

Implementation wise: 
Updated the HMasterInterface to pass the calls onto the zookeeper.
Added a zk listener to handle updates from the master to the RS to backup.
Added a utility for removing files and finding archive directories
Added tests for the regionserver and catalogjanitor approaches.
Added creation of manager in regionserver.


This addresses bug HBASE-5547.
https://issues.apache.org/jira/browse/HBASE-5547


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/HConstants.java 2f432c4 
  src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableMonitor.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/backup/TableHFileArchiveTracker.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f 
  src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java e751f65 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 0ad9b18 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1f09be1 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java f7ac81a 
  src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 
6884d53 
  src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e 
  src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 33bc1d0 
  src/main/resources/hbase-default.xml 42d1c4e 
  src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java 9fba339 
  
src/test/java/org/apache/hadoop/hbase/backup/SimpleHFileArchiveTableMonitor.java
 PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchivingCleanup.java 
PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/backup/TestRegionDisposer.java 
PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 69ccc65 
  src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1020374 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java
 PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java 
PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 
3f61cfb 

Diff: https://reviews.apache.org/r/4633/diff


Testing
---

Added two tests for the separate cases - archiving via the regionserver and for 
the catalog tracker. Former runs in a mini cluster and also touches the changes 
to HMasterInterface and zookeeper.


Thanks,

Jesse



 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Attachments: java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, 
 java_HBASE-5547_v6.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS

[jira] [Commented] (HBASE-5045) Add the table name and cf name for the next call int the task monitor


[ 
https://issues.apache.org/jira/browse/HBASE-5045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272739#comment-13272739
 ] 

Zhihong Yu commented on HBASE-5045:
---

The next() call is no longer in HRegionServer.java for trunk.
See:
{code}
public Result next() throws IOException {
src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java
{code}

 Add the table name and cf name for the next call int the task monitor
 -

 Key: HBASE-5045
 URL: https://issues.apache.org/jira/browse/HBASE-5045
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Amir Shimoni
 Attachments: D2913.1.patch, D2913.2.patch, D3045.1.patch, 
 D3045.2.patch, D3045.3.patch


 In the task monitor, we don't have much information about the next call 
 compared to other operations.
 It would be nice to add the table name and cf name for each next call in the 
 task monitor.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272756#comment-13272756
 ] 

Zhihong Yu commented on HBASE-5547:
---

@Jesse:
Do you want to post latest patch here for Hadoop QA ?

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Attachments: java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, 
 java_HBASE-5547_v6.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5547) Don't delete HFiles when in backup mode


 [ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesse Yates updated HBASE-5547:
---

Attachment: java_HBASE-5547_v7.patch

Attaching latest version for a QA run.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Attachments: java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, 
 java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5984) TestLogRolling.testLogRollOnPipelineRestart failed with HADOOP 2.0.0


[ 
https://issues.apache.org/jira/browse/HBASE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272800#comment-13272800
 ] 

Jimmy Xiang commented on HBASE-5984:


Looked into it and found it is really a flaky one.  The issue is that it is too 
flaky.  It always fails on me with HADOOP 2.0.0.

 TestLogRolling.testLogRollOnPipelineRestart failed with HADOOP 2.0.0
 

 Key: HBASE-5984
 URL: https://issues.apache.org/jira/browse/HBASE-5984
 Project: HBase
  Issue Type: Test
  Components: test
Affects Versions: 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: hbase_5984.patch


 java.io.IOException: Cannot obtain block length for 
 LocatedBlock{BP-1455809779-127.0.0.1-1336670196362:blk_-6960847342982670493_1028;
  getBlockSize()=1474; corrupt=false; offset=0; locs=[127.0.0.1:58343, 
 127.0.0.1:48427]}
   at 
 org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:232)
   at 
 org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:177)
   at 
 org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:119)
   at org.apache.hadoop.hdfs.DFSInputStream.init(DFSInputStream.java:112)
   at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:928)
   at 
 org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:212)
   at 
 org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:75)
   at 
 org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1768)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.openFile(SequenceFileLogReader.java:66)
   at 
 org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1688)
   at 
 org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1709)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.init(SequenceFileLogReader.java:58)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:166)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:659)
   at 
 org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnPipelineRestart(TestLogRolling.java:498)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
   at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
   at 
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
   at 
 org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
   at 
 org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
   at 
 org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
   at 
 org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
   at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
   at 
 org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
   at 
 org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:

[jira] [Updated] (HBASE-5984) TestLogRolling.testLogRollOnPipelineRestart failed with HADOOP 2.0.0


 [ 
https://issues.apache.org/jira/browse/HBASE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-5984:
---

Attachment: hbase_5984.patch

 TestLogRolling.testLogRollOnPipelineRestart failed with HADOOP 2.0.0
 

 Key: HBASE-5984
 URL: https://issues.apache.org/jira/browse/HBASE-5984
 Project: HBase
  Issue Type: Test
  Components: test
Affects Versions: 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: hbase_5984.patch


 java.io.IOException: Cannot obtain block length for 
 LocatedBlock{BP-1455809779-127.0.0.1-1336670196362:blk_-6960847342982670493_1028;
  getBlockSize()=1474; corrupt=false; offset=0; locs=[127.0.0.1:58343, 
 127.0.0.1:48427]}
   at 
 org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:232)
   at 
 org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:177)
   at 
 org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:119)
   at org.apache.hadoop.hdfs.DFSInputStream.init(DFSInputStream.java:112)
   at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:928)
   at 
 org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:212)
   at 
 org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:75)
   at 
 org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1768)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.openFile(SequenceFileLogReader.java:66)
   at 
 org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1688)
   at 
 org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1709)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.init(SequenceFileLogReader.java:58)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:166)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:659)
   at 
 org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnPipelineRestart(TestLogRolling.java:498)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
   at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
   at 
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
   at 
 org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
   at 
 org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
   at 
 org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
   at 
 org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
   at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
   at 
 org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
   at 
 org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5984) TestLogRolling.testLogRollOnPipelineRestart failed with HADOOP 2.0.0


 [ 
https://issues.apache.org/jira/browse/HBASE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-5984:
---

Status: Patch Available  (was: Open)

Put up a patch to disable this test for now.

 TestLogRolling.testLogRollOnPipelineRestart failed with HADOOP 2.0.0
 

 Key: HBASE-5984
 URL: https://issues.apache.org/jira/browse/HBASE-5984
 Project: HBase
  Issue Type: Test
  Components: test
Affects Versions: 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: hbase_5984.patch


 java.io.IOException: Cannot obtain block length for 
 LocatedBlock{BP-1455809779-127.0.0.1-1336670196362:blk_-6960847342982670493_1028;
  getBlockSize()=1474; corrupt=false; offset=0; locs=[127.0.0.1:58343, 
 127.0.0.1:48427]}
   at 
 org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:232)
   at 
 org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:177)
   at 
 org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:119)
   at org.apache.hadoop.hdfs.DFSInputStream.init(DFSInputStream.java:112)
   at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:928)
   at 
 org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:212)
   at 
 org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:75)
   at 
 org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1768)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.openFile(SequenceFileLogReader.java:66)
   at 
 org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1688)
   at 
 org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1709)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.init(SequenceFileLogReader.java:58)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:166)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:659)
   at 
 org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnPipelineRestart(TestLogRolling.java:498)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
   at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
   at 
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
   at 
 org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
   at 
 org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
   at 
 org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
   at 
 org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
   at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
   at 
 org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
   at 
 org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5975) Failed suppression of fs shutdown hook with Hadoop 2.0.0

2012-05-10 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272809#comment-13272809
 ] 

Hudson commented on HBASE-5975:
---

Integrated in HBase-TRUNK #2861 (See 
[https://builds.apache.org/job/HBase-TRUNK/2861/])
HBASE-5975 Failed suppression of fs shutdown hook with Hadoop 2.0.0 (Jimmy 
Xiang) (Revision 1336875)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java


 Failed suppression of fs shutdown hook with Hadoop 2.0.0
 

 Key: HBASE-5975
 URL: https://issues.apache.org/jira/browse/HBASE-5975
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0, 0.94.1

 Attachments: 5975+5966.patch, hbase-5975.patch


 Unit test failed with error:  Failed suppression of fs shutdown hook
 ShutdownHookManager.deleteShutdownHook failed to delete the hook since it 
 should be runnable instead of a thread for HADOOP 2.0.0.
 For other HADOOP version, runnable should work too since thread implements 
 runnable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5984) TestLogRolling.testLogRollOnPipelineRestart failed with HADOOP 2.0.0


[ 
https://issues.apache.org/jira/browse/HBASE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272816#comment-13272816
 ] 

Zhihong Yu commented on HBASE-5984:
---

This test is useful - though flaky.

We can detect the hadoop version in test util and return early from this test 
if the version is beyond x.y

 TestLogRolling.testLogRollOnPipelineRestart failed with HADOOP 2.0.0
 

 Key: HBASE-5984
 URL: https://issues.apache.org/jira/browse/HBASE-5984
 Project: HBase
  Issue Type: Test
  Components: test
Affects Versions: 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: hbase_5984.patch


 java.io.IOException: Cannot obtain block length for 
 LocatedBlock{BP-1455809779-127.0.0.1-1336670196362:blk_-6960847342982670493_1028;
  getBlockSize()=1474; corrupt=false; offset=0; locs=[127.0.0.1:58343, 
 127.0.0.1:48427]}
   at 
 org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:232)
   at 
 org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:177)
   at 
 org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:119)
   at org.apache.hadoop.hdfs.DFSInputStream.init(DFSInputStream.java:112)
   at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:928)
   at 
 org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:212)
   at 
 org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:75)
   at 
 org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1768)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.openFile(SequenceFileLogReader.java:66)
   at 
 org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1688)
   at 
 org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1709)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.init(SequenceFileLogReader.java:58)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:166)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:659)
   at 
 org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnPipelineRestart(TestLogRolling.java:498)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
   at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
   at 
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
   at 
 org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
   at 
 org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
   at 
 org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
   at 
 org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
   at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
   at 
 org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
   at 
 org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:

[jira] [Commented] (HBASE-5975) Failed suppression of fs shutdown hook with Hadoop 2.0.0

2012-05-10 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272819#comment-13272819
 ] 

Hudson commented on HBASE-5975:
---

Integrated in HBase-0.94 #187 (See 
[https://builds.apache.org/job/HBase-0.94/187/])
HBASE-5975 Failed suppression of fs shutdown hook with Hadoop 2.0.0 (Jimmy 
Xiang) (Revision 1336876)

 Result = SUCCESS
tedyu : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java


 Failed suppression of fs shutdown hook with Hadoop 2.0.0
 

 Key: HBASE-5975
 URL: https://issues.apache.org/jira/browse/HBASE-5975
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0, 0.94.1

 Attachments: 5975+5966.patch, hbase-5975.patch


 Unit test failed with error:  Failed suppression of fs shutdown hook
 ShutdownHookManager.deleteShutdownHook failed to delete the hook since it 
 should be runnable instead of a thread for HADOOP 2.0.0.
 For other HADOOP version, runnable should work too since thread implements 
 runnable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5985) TestMetaMigrationRemovingHTD failed with HADOOP 2.0.0


[ 
https://issues.apache.org/jira/browse/HBASE-5985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272841#comment-13272841
 ] 

Jimmy Xiang commented on HBASE-5985:


These two tests are not touched by this patch, and they are green on my box.

 TestMetaMigrationRemovingHTD failed with HADOOP 2.0.0
 -

 Key: HBASE-5985
 URL: https://issues.apache.org/jira/browse/HBASE-5985
 Project: HBase
  Issue Type: Test
  Components: test
Affects Versions: 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: hbase-5985.patch


 ---
 Test set: org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD
 ---
 Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.448 sec  
 FAILURE!
 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD  Time elapsed: 0 
 sec   ERROR!
 java.io.IOException: Failed put; errcode=1
 at 
 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD.doFsCommand(TestMetaMigrationRemovingHTD.java:124)
 at 
 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD.setUpBeforeClass(TestMetaMigrationRemovingHTD.java:80)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at 
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
 at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
 at 
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
 at 
 org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
 at 
 org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
 at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
 at 
 org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
 at 
 org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5984) TestLogRolling.testLogRollOnPipelineRestart failed with HADOOP 2.0.0


[ 
https://issues.apache.org/jira/browse/HBASE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272842#comment-13272842
 ] 

Jimmy Xiang commented on HBASE-5984:


Sure, it is fine with me.

 TestLogRolling.testLogRollOnPipelineRestart failed with HADOOP 2.0.0
 

 Key: HBASE-5984
 URL: https://issues.apache.org/jira/browse/HBASE-5984
 Project: HBase
  Issue Type: Test
  Components: test
Affects Versions: 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: hbase_5984.patch


 java.io.IOException: Cannot obtain block length for 
 LocatedBlock{BP-1455809779-127.0.0.1-1336670196362:blk_-6960847342982670493_1028;
  getBlockSize()=1474; corrupt=false; offset=0; locs=[127.0.0.1:58343, 
 127.0.0.1:48427]}
   at 
 org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:232)
   at 
 org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:177)
   at 
 org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:119)
   at org.apache.hadoop.hdfs.DFSInputStream.init(DFSInputStream.java:112)
   at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:928)
   at 
 org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:212)
   at 
 org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:75)
   at 
 org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1768)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.openFile(SequenceFileLogReader.java:66)
   at 
 org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1688)
   at 
 org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1709)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.init(SequenceFileLogReader.java:58)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:166)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:659)
   at 
 org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnPipelineRestart(TestLogRolling.java:498)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
   at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
   at 
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
   at 
 org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
   at 
 org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
   at 
 org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
   at 
 org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
   at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
   at 
 org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
   at 
 org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
   at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5973) Add ability for potentially long-running IPC calls to abort if client disconnects

2012-05-10 Thread stack (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272845#comment-13272845
]

stack commented on HBASE-5973:
--

Might Todd. When we call this:

{code}
private boolean nextInternal(int limit, String metric) throws IOException {
+ RpcCallContext rpcCall = HBaseServer.getCurrentCall();
while (true) {
+if (rpcCall != null) {
+ // If a user specifies a too-restrictive or too-slow scanner, the
+ // client might time out and disconnect while the server side
+ // is still processing the request. We should abort aggressively
+ // in that case.
+ rpcCall.throwExceptionIfCallerDisconnected();
+}
{code}

... if connection is closed when we check, the exception does not come out here
and abort this current nextInternal invocation? Rather, it comes out on the
stuck handler? Does it interrupt the ongoing call, the one w/o a client?

Pardon dumb question. Just trying to understand how this fix works.

Add ability for potentially long-running IPC calls to abort if client
disconnects
-

Key: HBASE-5973
URL: https://issues.apache.org/jira/browse/HBASE-5973
Project: HBase
Issue Type: Improvement
Components: ipc
Affects Versions: 0.90.7, 0.92.1, 0.94.0, 0.96.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Fix For: 0.92.2, 0.96.0, 0.94.1

Attachments: hbase-5973-0.92.txt, hbase-5973-0.94.txt,
hbase-5973-0.94.txt, hbase-5973.txt, hbase-5973.txt, hbase-5973.txt

We recently had a cluster issue where a user was submitting scanners with a
very restrictive filter, and then calling next() with a high scanner caching
value. The clients would generally time out the next() call and disconnect,
but the IPC kept running looking to fill the requested number of rows. Since
this was in the context of MR, the tasks making the calls would retry, and
the retries wuld be more likely to time out due to contention with the
previous still-running scanner next() call. Eventually, the system spiraled
out of control.
We should add a hook to the IPC system so that RPC calls can check if the
client has already disconnected. In such a case, the next() call could abort
processing, given any further work is wasted. I imagine coprocessor
endpoints, etc, could make good use of this as well.

[jira] [Commented] (HBASE-5966) MapReduce based tests broken on Hadoop 2.0.0-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-5966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272844#comment-13272844
 ] 

Jimmy Xiang commented on HBASE-5966:


These tests are green on my box.

 MapReduce based tests broken on Hadoop 2.0.0-alpha
 --

 Key: HBASE-5966
 URL: https://issues.apache.org/jira/browse/HBASE-5966
 Project: HBase
  Issue Type: Bug
  Components: mapred, mapreduce, test
Affects Versions: 0.94.0, 0.96.0
 Environment: Hadoop 2.0.0-alpha-SNAPSHOT, HBase 0.94.0-SNAPSHOT, 
 Ubuntu 12.04 LTS (GNU/Linux 3.2.0-24-generic x86_64)
Reporter: Andrew Purtell
Assignee: Jimmy Xiang
 Attachments: HBASE-5966-1.patch, HBASE-5966.patch, hbase-5966.patch


 Some fairly recent change in Hadoop 2.0.0-alpha has broken our MapReduce test 
 rigging. Below is a representative error, can be easily reproduced with:
 {noformat}
 mvn -PlocalTests -Psecurity \
   -Dhadoop.profile=23 -Dhadoop.version=2.0.0-SNAPSHOT \
   clean test \
   -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
 {noformat}
 And the result:
 {noformat}
 ---
  T E S T S
 ---
 Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
 Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 54.292 sec 
  FAILURE!
 ---
 Test set: org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
 ---
 Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 54.292 sec 
  FAILURE!
 testMultiRegionTable(org.apache.hadoop.hbase.mapreduce.TestTableMapReduce)  
 Time elapsed: 21.935 sec   ERROR!
 java.lang.reflect.UndeclaredThrowableException
   at 
 org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:135)
   at 
 org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:134)
   at 
 org.apache.hadoop.mapred.ResourceMgrDelegate.getNewJobID(ResourceMgrDelegate.java:183)
   at org.apache.hadoop.mapred.YARNRunner.getNewJobID(YARNRunner.java:216)
   at 
 org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:339)
   at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1226)
   at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1223)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:416)
   at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
   at org.apache.hadoop.mapreduce.Job.submit(Job.java:1223)
   at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1244)
   at 
 org.apache.hadoop.hbase.mapreduce.TestTableMapReduce.runTestOnTable(TestTableMapReduce.java:151)
   at 
 org.apache.hadoop.hbase.mapreduce.TestTableMapReduce.testMultiRegionTable(TestTableMapReduce.java:129)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:616)
   at 
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
   at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
   at 
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
   at 
 org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
   at org.junit.rules.RunRules.evaluate(RunRules.java:18)
   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
   at 
 org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
   at 
 org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
   at

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

2012-05-10 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272848#comment-13272848
 ] 

jirapos...@reviews.apache.org commented on HBASE-5547:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4633/#review7794
---


Heading to a meeting.
Below is first part of review.


src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java
https://reviews.apache.org/r/4633/#comment17105

We know all is true when table is null, right ?

Seems we don't need this boolean.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java
https://reviews.apache.org/r/4633/#comment17100

aren't - are



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java
https://reviews.apache.org/r/4633/#comment17101

Either remove 'down' or move it to the end of the sentence



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java
https://reviews.apache.org/r/4633/#comment17102

Please check return value.

I think we should remember the files which we are unable to delete and 
display them at the end.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java
https://reviews.apache.org/r/4633/#comment17104

Would anyone specify start == end ?



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableMonitor.java
https://reviews.apache.org/r/4633/#comment17106

License, please.



src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java
https://reviews.apache.org/r/4633/#comment17107

License.



src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java
https://reviews.apache.org/r/4633/#comment17108

This condition deserves an exception, I think.



src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java
https://reviews.apache.org/r/4633/#comment17109

Is it possible to provide more information in this exception ?
e.g. by using toArchive collection ?


- Ted


On 2012-05-10 20:28:50, Jesse Yates wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4633/
bq.  ---
bq.  
bq.  (Updated 2012-05-10 20:28:50)
bq.  
bq.  
bq.  Review request for hbase, Michael Stack and Lars Hofhansl.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Essentially, whenever an hfile would be deleted, it is instead moved to 
the archive directory. In this impl, the archive directory is on a per table 
basis, but defaults to '.archive'. Removing hfiles occurs in three places - 
compaction, merge and catalog janitor. The former and two latter are distinctly 
different code paths, but but did pull out some similarities. The latter two 
end up calling the same method, so there should be a reasonable amount of 
overlap.
bq.  
bq.  Implementation wise: 
bq.  Updated the HMasterInterface to pass the calls onto the zookeeper.
bq.  Added a zk listener to handle updates from the master to the RS to 
backup.
bq.  Added a utility for removing files and finding archive directories
bq.  Added tests for the regionserver and catalogjanitor approaches.
bq.  Added creation of manager in regionserver.
bq.  
bq.  
bq.  This addresses bug HBASE-5547.
bq.  https://issues.apache.org/jira/browse/HBASE-5547
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java 2f432c4 
bq.src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java 
PRE-CREATION 
bq.
src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableMonitor.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java 
PRE-CREATION 
bq.
src/main/java/org/apache/hadoop/hbase/backup/TableHFileArchiveTracker.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f 
bq.src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java e751f65 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 0ad9b18 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1f09be1 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
f7ac81a 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 
6884d53 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e 
bq.src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java 
PRE-CREATION 
bq.

[jira] [Commented] (HBASE-5973) Add ability for potentially long-running IPC calls to abort if client disconnects

[
https://issues.apache.org/jira/browse/HBASE-5973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272852#comment-13272852
]

Todd Lipcon commented on HBASE-5973:

Sorry, Stack.. I don't follow your question. If getCurrentCall() is non-null,
that implies that we are within one of the handler threads. So, throwing the
exception here will abort the next() call in that same handler.

Add ability for potentially long-running IPC calls to abort if client
disconnects
-

Attachments: hbase-5973-0.92.txt, hbase-5973-0.94.txt,
hbase-5973-0.94.txt, hbase-5973.txt, hbase-5973.txt, hbase-5973.txt

[jira] [Updated] (HBASE-5342) Grant/Revoke global permissions

2012-05-10 Thread Matteo Bertozzi (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-5342:
---

Attachment: HBASE-5342-v3.patch

Keep the old protocol version
mark old grant/revoke as deprecated
catch if server doesn't support new grant/revoke with global and fallback to 
the old method if necessary.

 Grant/Revoke global permissions
 ---

 Key: HBASE-5342
 URL: https://issues.apache.org/jira/browse/HBASE-5342
 Project: HBase
  Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Matteo Bertozzi
 Attachments: HBASE-5342-draft.patch, HBASE-5342-v0.patch, 
 HBASE-5342-v1.patch, HBASE-5342-v2.patch, HBASE-5342-v3.patch


 HBASE-3025 introduced simple ACLs based on coprocessors. It defines 
 global/table/cf/cq level permissions. However, there is no way to 
 grant/revoke global level permissions, other than the hbase.superuser conf 
 setting. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5984) TestLogRolling.testLogRollOnPipelineRestart failed with HADOOP 2.0.0


[ 
https://issues.apache.org/jira/browse/HBASE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272866#comment-13272866
 ] 

Hadoop QA commented on HBASE-5984:
--

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12526432/hbase_5984.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1839//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1839//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1839//console

This message is automatically generated.

 TestLogRolling.testLogRollOnPipelineRestart failed with HADOOP 2.0.0
 

 Key: HBASE-5984
 URL: https://issues.apache.org/jira/browse/HBASE-5984
 Project: HBase
  Issue Type: Test
  Components: test
Affects Versions: 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: 0.96.0

 Attachments: hbase_5984.patch


 java.io.IOException: Cannot obtain block length for 
 LocatedBlock{BP-1455809779-127.0.0.1-1336670196362:blk_-6960847342982670493_1028;
  getBlockSize()=1474; corrupt=false; offset=0; locs=[127.0.0.1:58343, 
 127.0.0.1:48427]}
   at 
 org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:232)
   at 
 org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:177)
   at 
 org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:119)
   at org.apache.hadoop.hdfs.DFSInputStream.init(DFSInputStream.java:112)
   at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:928)
   at 
 org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:212)
   at 
 org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:75)
   at 
 org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1768)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.openFile(SequenceFileLogReader.java:66)
   at 
 org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1688)
   at 
 org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1709)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.init(SequenceFileLogReader.java:58)
   at 
 org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:166)
   at 
 org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:659)
   at 
 org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnPipelineRestart(TestLogRolling.java:498)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at 
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
   at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
   at 
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
   at 
 org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
   at 
 org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
   at 
 org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
   at

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272882#comment-13272882
]

Hadoop QA commented on HBASE-5547:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12526422/java_HBASE-5547_v7.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 22 new or modified tests.

+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.TestDrainingServer

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1840//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1840//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1840//console

This message is automatically generated.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Attachments: java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch,
java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch

This came up in a discussion I had with Stack.
It would be nice if HBase could be notified that a backup is in progress (via
a znode for example) and in that case either:
1. rename HFiles to be delete to file.bck
2. rename the HFiles into a special directory
3. rename them to a general trash directory (which would not need to be tied
to backup mode).
That way it should be able to get a consistent backup based on HFiles (HDFS
snapshots or hard links would be better options here, but we do not have
those).
#1 makes cleanup a bit harder.

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

2012-05-10 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272885#comment-13272885
 ] 

jirapos...@reviews.apache.org commented on HBASE-5732:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4953/
---

(Updated 2012-05-10 23:19:45.031886)


Review request for Ted Yu, Michael Stack and Andrew Purtell.


Changes
---

Upon testing with with a mapreduce job (PerformanceEvaluation.java), I found a 
bug to do with delegation tokens. This patch fixes that. 


Summary
---

Reviewboard request for HBASE-5732


This addresses bug HBASE-5732.
https://issues.apache.org/jira/browse/HBASE-5732


Diffs (updated)
-

  http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1336441 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/AdminProtocol.java
 1336441 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClientProtocol.java
 1336441 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java
 1336441 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
 1336441 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1336441 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/RegionServerStatusProtocol.java
 1336441 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java
 1336441 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java
 1336441 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java
 1336441 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/TablePermission.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java
 PRE-CREATION

[jira] [Updated] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

2012-05-10 Thread Devaraj Das (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj Das updated HBASE-5732:
---

Attachment: rpcengine-merge.12.patch

This is the patch that I just uploaded on RB. The patch still applies. Would 
appreciate a review/commit round on this :-)

Ted, yes, I'll look for security fixes over in Hadoop that HBase should be 
caring about. Thanks for the pointers.

 Remove the SecureRPCEngine and merge the security-related logic in the core 
 engine
 --

 Key: HBASE-5732
 URL: https://issues.apache.org/jira/browse/HBASE-5732
 Project: HBase
  Issue Type: Improvement
Reporter: Devaraj Das
Assignee: Devaraj Das
 Attachments: 5732-rpcengine-merge.11.patch, 
 5732-rpcengine-merge.7.patch, rpcengine-merge.10.patch, 
 rpcengine-merge.11.patch, rpcengine-merge.12.patch, rpcengine-merge.3.patch, 
 rpcengine-merge.4.patch, rpcengine-merge.9.patch, rpcengine-merge.patch


 Remove the SecureRPCEngine and merge the security-related logic in the core 
 engine. Follow up to HBASE-5727.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5954) Allow proper fsync support for HBase


 [ 
https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5954:
-

Attachment: 5954-trunk-hdfs-trunk.txt

Here's a patch against HBase-trunk matching the v2-trunk patch in HDFS-744.

Again, this is not yet configurable and just for testing.


 Allow proper fsync support for HBase
 

 Key: HBASE-5954
 URL: https://issues.apache.org/jira/browse/HBASE-5954
 Project: HBase
  Issue Type: Improvement
Reporter: Lars Hofhansl
 Attachments: 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5986) Clients can see holes in the META table when regions are being split

Enis Soztutar created HBASE-5986:


 Summary: Clients can see holes in the META table when regions are 
being split
 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar


We found this issue when running large scale ingestion tests for HBASE-5754. 
The problem is that the .META. table updates are not atomic while splitting a 
region. In SplitTransaction, there is a time lap between the marking the parent 
offline, and adding of daughters to the META table. This can result in clients 
using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) 
missing regions which are made just offline, but the daughters are not added 
yet. 

This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5986) Clients can see holes in the META table when regions are being split


 [ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-5986:
-

Attachment: HBASE-5986-test_v1.patch

Attaching a unit test to illustrate the problem. The test fails for me when 
splitting the first region, the client sees 0 regions for some time. 

 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Attachments: HBASE-5986-test_v1.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5342) Grant/Revoke global permissions

[
https://issues.apache.org/jira/browse/HBASE-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272915#comment-13272915
]

Hadoop QA commented on HBASE-5342:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12526440/HBASE-5342-v3.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 9 new or modified tests.

+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.TestRegionRebalancing
org.apache.hadoop.hbase.TestDrainingServer
org.apache.hadoop.hbase.coprocessor.TestClassLoading

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1841//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1841//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1841//console

This message is automatically generated.

Grant/Revoke global permissions
---

Key: HBASE-5342
URL: https://issues.apache.org/jira/browse/HBASE-5342
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Matteo Bertozzi
Attachments: HBASE-5342-draft.patch, HBASE-5342-v0.patch,
HBASE-5342-v1.patch, HBASE-5342-v2.patch, HBASE-5342-v3.patch

HBASE-3025 introduced simple ACLs based on coprocessors. It defines
global/table/cf/cq level permissions. However, there is no way to
grant/revoke global level permissions, other than the hbase.superuser conf
setting.

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272918#comment-13272918
]

Enis Soztutar commented on HBASE-5986:
--

Possible fixes I can think of:
1. Keep MetaScanner/MetaReader as non-consistent (as it is), but allow for a
consistent view for getting table regions. Since single row puts are atomic,
when the parent region is mutated to be offline, the HRI for daughters are
added to the row. So on MetaScanner.allTableRegions and similar calls, we can
keep track of daughter regions from split parents and return them to the
client.
2. Make MetaScanner consistent, in that, whenever it sees a split parent, it
blocks until the daughters are available.
3. We have region-local transactions now, so if we ensure that the rows for
parent and daughters will be served from the same META region, then we can
update all three rows atomically. Maybe we can come up with a META-specific
split policy to ensure split-regions go to the same META region.
Thoughts?

Clients can see holes in the META table when regions are being split

Key: HBASE-5986
URL: https://issues.apache.org/jira/browse/HBASE-5986
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Attachments: HBASE-5986-test_v1.patch

We found this issue when running large scale ingestion tests for HBASE-5754.
The problem is that the .META. table updates are not atomic while splitting a
region. In SplitTransaction, there is a time lap between the marking the
parent offline, and adding of daughters to the META table. This can result in
clients using MetaScanner, of HTable.getStartEndKeys (used by the
TableInputFormat) missing regions which are made just offline, but the
daughters are not added yet.
This is also related to HBASE-4335.

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272948#comment-13272948
]

Lars Hofhansl commented on HBASE-5986:
--

I think we are assuming in many other places that META only has a single region.

Is there another alternative, such as adding the daughter regions first, and
then have HTable disentangle conflicts?

Clients can see holes in the META table when regions are being split

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

[
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272947#comment-13272947
]

Hadoop QA commented on HBASE-5732:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12526450/rpcengine-merge.12.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 37 new or modified tests.

+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 27 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1842//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1842//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1842//console

This message is automatically generated.

Remove the SecureRPCEngine and merge the security-related logic in the core
engine
--

Key: HBASE-5732
URL: https://issues.apache.org/jira/browse/HBASE-5732
Project: HBase
Issue Type: Improvement
Reporter: Devaraj Das
Assignee: Devaraj Das
Attachments: 5732-rpcengine-merge.11.patch,
5732-rpcengine-merge.7.patch, rpcengine-merge.10.patch,
rpcengine-merge.11.patch, rpcengine-merge.12.patch, rpcengine-merge.3.patch,
rpcengine-merge.4.patch, rpcengine-merge.9.patch, rpcengine-merge.patch

Remove the SecureRPCEngine and merge the security-related logic in the core
engine. Follow up to HBASE-5727.

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272952#comment-13272952
]

Enis Soztutar commented on HBASE-5986:
--

bq. I think we are assuming in many other places that META only has a single
region.
The ROOT is -by-design- one region, but META is not, right?

bq. Is there another alternative, such as adding the daughter regions first,
and then have HTable disentangle conflicts?
I have thought about this as well, but then there is a time window in which you
have both the parent and daughter regions online, and parent not marked as
split. So the client again has to resolve that the returned regions are
overlapping.

Clients can see holes in the META table when regions are being split

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

2012-05-10 Thread Devaraj Das (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272960#comment-13272960
 ] 

Devaraj Das commented on HBASE-5732:


There are actually no unit test failures (when i look at the console). Not sure 
why hadoopqa gave a -1. 

 Remove the SecureRPCEngine and merge the security-related logic in the core 
 engine
 --

 Key: HBASE-5732
 URL: https://issues.apache.org/jira/browse/HBASE-5732
 Project: HBase
  Issue Type: Improvement
Reporter: Devaraj Das
Assignee: Devaraj Das
 Attachments: 5732-rpcengine-merge.11.patch, 
 5732-rpcengine-merge.7.patch, rpcengine-merge.10.patch, 
 rpcengine-merge.11.patch, rpcengine-merge.12.patch, rpcengine-merge.3.patch, 
 rpcengine-merge.4.patch, rpcengine-merge.9.patch, rpcengine-merge.patch


 Remove the SecureRPCEngine and merge the security-related logic in the core 
 engine. Follow up to HBASE-5727.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5539) asynchbase PerformanceEvaluation

2012-05-10 Thread Otis Gospodnetic (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272964#comment-13272964
 ] 

Otis Gospodnetic commented on HBASE-5539:
-

@Benoit, have you succeeded in running the benchmarks?  We are considering 
using asyncbase and would love to see your numbers from the comparison.

 asynchbase PerformanceEvaluation
 

 Key: HBASE-5539
 URL: https://issues.apache.org/jira/browse/HBASE-5539
 Project: HBase
  Issue Type: New Feature
  Components: performance
Reporter: Benoit Sigoure
Assignee: Benoit Sigoure
Priority: Minor
  Labels: benchmark
 Attachments: 0001-asynchbase-PerformanceEvaluation.patch


 I plugged [asynchbase|https://github.com/stumbleupon/asynchbase] into 
 {{PerformanceEvaluation}}.  This enables testing asynchbase from 
 {{PerformanceEvaluation}} and comparing its performance to {{HTable}}.  Also 
 asynchbase doesn't come with any benchmark, so it was good that I was able to 
 plug it into {{PerformanceEvaluation}} relatively easily.
 I am in the processing of collecting results on a dev cluster running 0.92.1 
 and will publish them once they're ready.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272966#comment-13272966
]

Lars Hofhansl commented on HBASE-5986:
--

True, META is not single region by design, but effectively it is (not really
suggesting that we add more code that relies on this). I think you're right we
could add a split policy that ensures that parent and daughters are always in
the same region.

As for the alternative:
If the client has to recheck in these (rare) cases that seems OK, as long as
the client can reliably tell that it does have to recheck.

Clients can see holes in the META table when regions are being split

[jira] [Commented] (HBASE-5546) Master assigns region in the original region server when opening region failed

2012-05-10 Thread gaojinchao (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272971#comment-13272971
 ] 

gaojinchao commented on HBASE-5546:
---

+1, Good job!
 

 Master assigns region in the original region server when opening region 
 failed  
 

 Key: HBASE-5546
 URL: https://issues.apache.org/jira/browse/HBASE-5546
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.0
Reporter: gaojinchao
Assignee: Ashutosh Jindal
Priority: Minor
 Fix For: 0.96.0

 Attachments: hbase-5546.patch, hbase-5546_1.patch


 Master assigns region in the original region server when 
 RS_ZK_REGION_FAILED_OPEN envent was coming.
 Maybe we should choose other region server.
 [2012-03-07 10:14:21,750] [DEBUG] [main-EventThread] 
 [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
 transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
 region=c70e98bdca98a0657a56436741523053
 [2012-03-07 10:14:31,826] [DEBUG] [main-EventThread] 
 [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
 transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
 region=c70e98bdca98a0657a56436741523053
 [2012-03-07 10:14:41,903] [DEBUG] [main-EventThread] 
 [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
 transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
 region=c70e98bdca98a0657a56436741523053
 [2012-03-07 10:14:51,975] [DEBUG] [main-EventThread] 
 [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
 transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
 region=c70e98bdca98a0657a56436741523053
 [2012-03-07 10:15:02,056] [DEBUG] [main-EventThread] 
 [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
 transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
 region=c70e98bdca98a0657a56436741523053
 [2012-03-07 10:15:12,167] [DEBUG] [main-EventThread] 
 [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
 transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
 region=c70e98bdca98a0657a56436741523053
 [2012-03-07 10:15:22,231] [DEBUG] [main-EventThread] 
 [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
 transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
 region=c70e98bdca98a0657a56436741523053
 [2012-03-07 10:15:32,303] [DEBUG] [main-EventThread] 
 [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
 transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
 region=c70e98bdca98a0657a56436741523053
 [2012-03-07 10:15:42,375] [DEBUG] [main-EventThread] 
 [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
 transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
 region=c70e98bdca98a0657a56436741523053
 [2012-03-07 10:15:52,447] [DEBUG] [main-EventThread] 
 [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
 transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
 region=c70e98bdca98a0657a56436741523053
 [2012-03-07 10:16:02,528] [DEBUG] [main-EventThread] 
 [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
 transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
 region=c70e98bdca98a0657a56436741523053
 [2012-03-07 10:16:12,600] [DEBUG] [main-EventThread] 
 [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
 transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
 region=c70e98bdca98a0657a56436741523053
 [2012-03-07 10:16:22,676] [DEBUG] [main-EventThread] 
 [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
 transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
 region=c70e98bdca98a0657a56436741523053

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5987) HFileBlockIndex improvement

2012-05-10 Thread Liyin Tang (JIRA)

Liyin Tang created HBASE-5987:
-

 Summary: HFileBlockIndex improvement
 Key: HBASE-5987
 URL: https://issues.apache.org/jira/browse/HBASE-5987
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang


Recently we find out a performance problem that it is quite slow when multiple 
requests are reading the same block of data or index. 

From the profiling, one of the causes is the IdLock contention which has been 
addressed in HBASE-5898. 

Another issue is that the HFileScanner will keep asking the HFileBlockIndex 
about the data block location for each target key value during the scan 
process(reSeekTo), even though the target key value has already been in the 
current data block. This issue will cause certain index block very HOT, 
especially when it is a sequential scan.

To solve this issue, we propose the following solutions:

First, we propose to lookahead for one more block index so that the 
HFileScanner would know the start key value of next data block. So if the 
target key value for the scan(reSeekTo) is smaller than that start kv of next 
data block, it means the target key value has a very high possibility in the 
current data block (if not in current data block, then the start kv of next 
data block should be returned. +Indexing on the start key has some defects 
here+) and it shall NOT query the HFileBlockIndex in this case. On the 
contrary, if the target key value is bigger, then it shall query the 
HFileBlockIndex. This improvement shall help to reduce the hotness of 
HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block 
Cache lookup.

Secondary, we propose to push this idea a little further that the 
HFileBlockIndex shall index on the last key value of each data block instead of 
indexing on the start key value. The motivation is to solve the HBASE-4443 
issue (avoid seeking to previous block when key you are interested in is the 
first one of a block) as well as +the defects mentioned above+.

For example, if the target key value is smaller than the start key value of 
the data block N. There is no way for sure the target key value is in the data 
block N or N-1. So it has to seek from data block N-1. However, if the block 
index is based on the last key value for each data block and the target key 
value is beween the last key value of data block N-1 and data block N, then the 
target key value is supposed be data block N for sure. 


As long as HBase only supports the forward scan, the last key value makes more 
sense to be indexed on than the start key value.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5987) HFileBlockIndex improvement

2012-05-10 Thread Liyin Tang (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Liyin Tang updated HBASE-5987:
--

Description:
Recently we find out a performance problem that it is quite slow when multiple
requests are reading the same block of data or index.

From the profiling, one of the causes is the IdLock contention which has been
addressed in HBASE-5898.

Another issue is that the HFileScanner will keep asking the HFileBlockIndex
about the data block location for each target key value during the scan
process(reSeekTo), even though the target key value has already been in the
current data block. This issue will cause certain index block very HOT,
especially when it is a sequential scan.

To solve this issue, we propose the following solutions:

First, we propose to lookahead for one more block index so that the
HFileScanner would know the start key value of next data block. So if the
target key value for the scan(reSeekTo) is smaller than that start kv of next
data block, it means the target key value has a very high possibility in the
current data block (if not in current data block, then the start kv of next
data block should be returned. +Indexing on the start key has some defects
here+) and it shall NOT query the HFileBlockIndex in this case. On the
contrary, if the target key value is bigger, then it shall query the
HFileBlockIndex. This improvement shall help to reduce the hotness of
HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block
Cache lookup.

Secondary, we propose to push this idea a little further that the
HFileBlockIndex shall index on the last key value of each data block instead of
indexing on the start key value. The motivation is to solve the HBASE-4443
issue (avoid seeking to previous block when key you are interested in is the
first one of a block) as well as +the defects mentioned above+.

For example, if the target key value is smaller than the start key value of
the data block N. There is no way for sure the target key value is in the data
block N or N-1. So it has to seek from data block N-1. However, if the block
index is based on the last key value for each data block and the target key
value is beween the last key value of data block N-1 and data block N, then the
target key value is supposed be data block N for sure.

As long as HBase only supports the forward scan, the last key value makes more
sense to be indexed on than the start key value.

Thanks Kannan and Mikhail for the insightful discussions and suggestions.

was:
Recently we find out a performance problem that it is quite slow when multiple
requests are reading the same block of data or index.

From the profiling, one of the causes is the IdLock contention which has been
addressed in HBASE-5898.

To solve this issue, we propose the following solutions:

As long as HBase only supports the forward scan, the last key value makes more
sense to be indexed on than the start key value.

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split


[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272974#comment-13272974
 ] 

Zhihong Yu commented on HBASE-5986:
---

Can MetaScanner.allTableRegions() return special exception so that client knows 
it needs to check back ?

 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Attachments: HBASE-5986-test_v1.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine


[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272975#comment-13272975
 ] 

Zhihong Yu commented on HBASE-5732:
---

Reattaching patch v12 would allow Hadoop QA to rerun the tests.

We can also run test suite ourselves and observe if there is any hanging unit 
test(s).

 Remove the SecureRPCEngine and merge the security-related logic in the core 
 engine
 --

 Key: HBASE-5732
 URL: https://issues.apache.org/jira/browse/HBASE-5732
 Project: HBase
  Issue Type: Improvement
Reporter: Devaraj Das
Assignee: Devaraj Das
 Attachments: 5732-rpcengine-merge.11.patch, 
 5732-rpcengine-merge.7.patch, rpcengine-merge.10.patch, 
 rpcengine-merge.11.patch, rpcengine-merge.12.patch, rpcengine-merge.3.patch, 
 rpcengine-merge.4.patch, rpcengine-merge.9.patch, rpcengine-merge.patch


 Remove the SecureRPCEngine and merge the security-related logic in the core 
 engine. Follow up to HBASE-5727.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine


 [ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5732:
--

Attachment: 5732-rpcengine-merge.12.patch

Re-attaching patch v12.

 Remove the SecureRPCEngine and merge the security-related logic in the core 
 engine
 --

 Key: HBASE-5732
 URL: https://issues.apache.org/jira/browse/HBASE-5732
 Project: HBase
  Issue Type: Improvement
Reporter: Devaraj Das
Assignee: Devaraj Das
 Attachments: 5732-rpcengine-merge.11.patch, 
 5732-rpcengine-merge.12.patch, 5732-rpcengine-merge.7.patch, 
 rpcengine-merge.10.patch, rpcengine-merge.11.patch, rpcengine-merge.12.patch, 
 rpcengine-merge.3.patch, rpcengine-merge.4.patch, rpcengine-merge.9.patch, 
 rpcengine-merge.patch


 Remove the SecureRPCEngine and merge the security-related logic in the core 
 engine. Follow up to HBASE-5727.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272987#comment-13272987
]

Enis Soztutar commented on HBASE-5986:
--

I have implemented approach 1 by adding split daughters to the returned map
from MetaScanner.allTableRegions(). But then the problem is that, we are
returning regions which does not yet exists in the META table, so any
subsequent getRegion call will fail.

Thinking a bit more about 3, I think we already guarantee that the region split
parent, and daughters fall into the same META region. Let's say we have two
regions region1 and region2, with start keys start_key*, and timestamps ts*
respectively.

Before split:
{code}
table start_key1 ts1 encoded_name1
table start_key2 ts2 encoded_name2
{code}

Now, if we split region1, daughters will be sorted after region1, and before
region2:
{code}
table start_key1 ts1 encoded_name1 offline split
table start_key1 ts3 encoded_name1
table mid_key1 ts3 encoded_name1
table start_key2 ts2 encoded_name2
{code}

we know this since we have the invariants ts3 ts1
(SplitTransaction.getDaughterRegionIdTimestamp()) and start_key1 mid_key1
start_key2. Even if we have a region boundary between start_key1 and start_key2
in the META table, the daughters will be co-located with the parent. The only
exception is that while the user table is split, we have a concurrent split for
the META table, and the new region boundary is chosen to be between the parent
and daughters. With some effort, we can prevent this, but it seems to be very
highly unlikely.

So, if my analysis is correct, that means option 3 seems like the best choice,
since this will not complicate the meta scan code. The problem is that, there
is no internal API to do multi-row transcations other than using the
coprocessor. Should we think of allowing that w/o coprocessors?

@Lars, does HRegion.mutateRowsWithLock() guarantee that a concurrent scanner
won't see partial changes?

Clients can see holes in the META table when regions are being split

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine

[
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272995#comment-13272995
]

Hadoop QA commented on HBASE-5732:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12526466/5732-rpcengine-merge.12.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 37 new or modified tests.

+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 27 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1843//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1843//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1843//console

This message is automatically generated.

Remove the SecureRPCEngine and merge the security-related logic in the core
engine
--

Key: HBASE-5732
URL: https://issues.apache.org/jira/browse/HBASE-5732
Project: HBase
Issue Type: Improvement
Reporter: Devaraj Das
Assignee: Devaraj Das
Attachments: 5732-rpcengine-merge.11.patch,
5732-rpcengine-merge.12.patch, 5732-rpcengine-merge.7.patch,
rpcengine-merge.10.patch, rpcengine-merge.11.patch, rpcengine-merge.12.patch,
rpcengine-merge.3.patch, rpcengine-merge.4.patch, rpcengine-merge.9.patch,
rpcengine-merge.patch

Remove the SecureRPCEngine and merge the security-related logic in the core
engine. Follow up to HBASE-5727.

[jira] [Commented] (HBASE-5987) HFileBlockIndex improvement

[
https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13273000#comment-13273000
]

Todd Lipcon commented on HBASE-5987:

Nice stuff Liyin. I was looking at scan performance a bit last week as well and
came to similar conclusions that the reseeks were pretty expensive for this
reason.

Another thing I noticed is that our CPU cache behavior is pretty bad when the
individual KVs are large. When I profiled L2 cache misses in oprofile, I saw a
bunch on the call to read the memstoreTS -- assumedly because it fell on a
different cache line than the rest of the KV. In my case, the KVs were just
over 128 bytes (2 cache lines), including their header fields, lengths etc. So
the access pattern looked like:

- hit cacheline 0 for kv0 header
- hit cacheline 2 for kv0 memstoreTS
- hit cacheline 0 repeatedly to do KV comparison
- hit cacheline 2 for kv1's header
- hit cacheline 4 for kv1's memstoreTS
- hit cacheline 2 for kv1 data comparison
etc.

For whatever reason, my CPU wasn't quite smart enough to kick prefetching in on
this access pattern. I tried recompiling JDK7 with the Unsafe.prefetchRead
intrinsic, but couldn't get any noticeable improvement with it. So I think for
better performance, we need some better in-memory layout for the HFile blocks,
so we can get O(lg n) reseeks instead of O(n), for example.

Have you seen the same?

HFileBlockIndex improvement
---

Key: HBASE-5987
URL: https://issues.apache.org/jira/browse/HBASE-5987
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang

Recently we find out a performance problem that it is quite slow when
multiple requests are reading the same block of data or index.
From the profiling, one of the causes is the IdLock contention which has been
addressed in HBASE-5898.
Another issue is that the HFileScanner will keep asking the HFileBlockIndex
about the data block location for each target key value during the scan
process(reSeekTo), even though the target key value has already been in the
current data block. This issue will cause certain index block very HOT,
especially when it is a sequential scan.
To solve this issue, we propose the following solutions:
First, we propose to lookahead for one more block index so that the
HFileScanner would know the start key value of next data block. So if the
target key value for the scan(reSeekTo) is smaller than that start kv of
next data block, it means the target key value has a very high possibility in
the current data block (if not in current data block, then the start kv of
next data block should be returned. +Indexing on the start key has some
defects here+) and it shall NOT query the HFileBlockIndex in this case. On
the contrary, if the target key value is bigger, then it shall query the
HFileBlockIndex. This improvement shall help to reduce the hotness of
HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block
Cache lookup.
Secondary, we propose to push this idea a little further that the
HFileBlockIndex shall index on the last key value of each data block instead
of indexing on the start key value. The motivation is to solve the HBASE-4443
issue (avoid seeking to previous block when key you are interested in is
the first one of a block) as well as +the defects mentioned above+.
For example, if the target key value is smaller than the start key value of
the data block N. There is no way for sure the target key value is in the
data block N or N-1. So it has to seek from data block N-1. However, if the
block index is based on the last key value for each data block and the target
key value is beween the last key value of data block N-1 and data block N,
then the target key value is supposed be data block N for sure.
As long as HBase only supports the forward scan, the last key value makes
more sense to be indexed on than the start key value.
Thanks Kannan and Mikhail for the insightful discussions and suggestions.

[jira] [Resolved] (HBASE-5951) Combining ColumnPrefixFilters using filterlist.

2012-05-10 Thread Anoop Sam John (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John resolved HBASE-5951.
---

Resolution: Not A Problem

This is not a defect.. 2 ColumnPrefixFilters were used with AND condition and 
that is why all the KVs were getting filtered out. For the usage scenario 
custom filter needs to be written.
Closing after the agreement from Davis

 Combining ColumnPrefixFilters using filterlist.
 ---

 Key: HBASE-5951
 URL: https://issues.apache.org/jira/browse/HBASE-5951
 Project: HBase
  Issue Type: Bug
  Components: filters
Affects Versions: 0.92.1
 Environment: Combining multiple ColumnPrefixFilters using filterlist 
 does not return exact result.
Reporter: Davis Abraham

 FilterList listOfFilters = new FilterList (FilterList.Operator.MUST_PASS_ALL);
 FilterList listOfFilters1 = new FilterList 
 (FilterList.Operator.MUST_PASS_ALL);
 FilterList listOfFilters2 = new FilterList 
 (FilterList.Operator.MUST_PASS_ALL);
 SingleColumnValueFilter SingleFilter1 = new
 SingleColumnValueFilter(Bytes.toBytes(cf),
 Bytes.toBytes(country), CompareOp.EQUAL,
 Bytes.toBytes(USA));
 listOfFilters.addFilter(SingleFilter1);
 ValueFilter VF1= new ValueFilter (CompareOp.EQUAL, 
 new SubstringComparator(ABC));
 ColumnPrefixFilter CF1= new ColumnPrefixFilter(Bytes.toBytes(name));
 listOfFilters1.addFilter(CF1);
 listOfFilters1.addFilter(VF1);
 listOfFilters.addFilter(listOfFilters1);
 ValueFilter VF2= new ValueFilter (CompareOp.EQUAL, 
 new SubstringComparator(ED));
 ColumnPrefixFilter CF2= new ColumnPrefixFilter (Bytes.toBytes(CRS));
 listOfFilters2.addFilter(CF2);
 listOfFilters2.addFilter(VF2);
 listOfFilters.addFilter(listOfFilters2);
 When i do a combibation of SingleFilter1 and listOfFilters1
  the result is correct, same way the combination of 
 SingleFilter1 and listOfFilters2 is returing correct result.
 But when all the three is combined im not getting any result..
 Is it the problem with multiple ColumnPrefixFilter??? 
 Value ABC exist in name.0 and value ED exist in CRS.0 and it is in
 the same row under same Column Family.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5828) TestLogRolling fails in 0.90 branch killing the test suite up on jenkins

2012-05-10 Thread xufeng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

xufeng updated HBASE-5828:
--

Attachment: HBASE-5828_90_v1_100times_test_result.txt
HBASE-5828_90_v1.patch

I create a patch and run this patch 100 times by shell.
{noformat}
for((i=1;i=100;i++));
do
   mvn clean -Dtest=TestLogRolling test  
HBASE-5828_90_v1_100times_test_result.txt
   mv target target_${i}
done
{noformat}
2 times（64th 75th）failed,I do not why happened now.
{noformat}
---
Test set: org.apache.hadoop.hbase.regionserver.wal.TestLogRolling
---
Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1,219.296 sec 
 FAILURE!
testLogRollOnPipelineRestart(org.apache.hadoop.hbase.regionserver.wal.TestLogRolling)
  Time elapsed: 952.159 sec   ERROR!
org.apache.hadoop.hbase.regionserver.wal.FailedLogCloseException: #1336683442040
at 
org.apache.hadoop.hbase.regionserver.wal.HLog.cleanupCurrentWriter(HLog.java:802)
at 
org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:560)
at 
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnPipelineRestart(TestLogRolling.java:476)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at 
org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:62)
at 
org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:140)
at 
org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:165)
at org.apache.maven.surefire.Surefire.run(Surefire.java:107)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:289)
at 
org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1005)
Caused by: java.io.IOException: Error Recovery for block 
blk_1933341499089292179_1016 failed  because recovery from primary datanode 
127.0.0.1:50920 failed 6 times.  Pipeline was 127.0.0.1:50920. Aborting...
at 
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2741)
at 
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1500(DFSClient.java:2172)
at 
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2371)
{noformat}

{noformat}
---
Test set: org.apache.hadoop.hbase.regionserver.wal.TestLogRolling
---
Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1,065.652 sec 
 FAILURE!
testLogRollOnPipelineRestart(org.apache.hadoop.hbase.regionserver.wal.TestLogRolling)
  Time elapsed: 830.246 sec   ERROR!
org.apache.hadoop.hbase.DroppedSnapshotException: region: 
TestLogRolling,,1336675047757.bfce863e5843952e14649dc6fc8a53dd.
at

[jira] [Updated] (HBASE-5959) Add other load balancers

2012-05-10 Thread Elliott Clark (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-5959:
-

Attachment: HBASE-5959-1.patch

Better version with factored tests.

It still needs some more documentation and I'm hoping to add more cost 
functions.  Though I might leave that for another issue.

 Add other load balancers
 

 Key: HBASE-5959
 URL: https://issues.apache.org/jira/browse/HBASE-5959
 Project: HBase
  Issue Type: New Feature
  Components: master
Reporter: Elliott Clark
Assignee: Elliott Clark
 Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch


 Now that balancers are pluggable we should give some options.b

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5987) HFileBlockIndex improvement

2012-05-10 Thread Liyin Tang (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13273031#comment-13273031
]

Liyin Tang commented on HBASE-5987:
---

Nice summary! I haven't profiled the CPU in details and shall try it :)

We found out this problem based on the following 2 experiments:
1) Single HBase client sequentially scans 8G of kvs from one region. Each kv is
approximately 50 Byte and all of them are 100% cached in block cache. It takes
approximately 4 mins to finish.
2) 20 HBase clients issue the same scan in parallel against the same set of 8G
cached kvs. The finish time varies from 20 min to 50 min.

In the region server, the network and disk are not busy and cpu usage increased
only by 1%.
And most of IPC threads for the next call waited for the IdLock to read
HFileBlockIndex.

HFileBlockIndex improvement
---

Key: HBASE-5987
URL: https://issues.apache.org/jira/browse/HBASE-5987
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang

[jira] [Commented] (HBASE-5973) Add ability for potentially long-running IPC calls to abort if client disconnects

2012-05-10 Thread stack (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13273033#comment-13273033
]

stack commented on HBASE-5973:
--

@Todd You answered my question (that and looking at code). I see how this
works now and how it can be a kill-switch particularly for the case where heavy
filtering has us passing on lots of rows.

+1 on the patch.

Add ability for potentially long-running IPC calls to abort if client
disconnects
-

Attachments: hbase-5973-0.92.txt, hbase-5973-0.94.txt,
hbase-5973-0.94.txt, hbase-5973.txt, hbase-5973.txt, hbase-5973.txt

[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine


[ 
https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13273036#comment-13273036
 ] 

Zhihong Yu commented on HBASE-5732:
---

Patch v12 is ready for integration.

 Remove the SecureRPCEngine and merge the security-related logic in the core 
 engine
 --

 Key: HBASE-5732
 URL: https://issues.apache.org/jira/browse/HBASE-5732
 Project: HBase
  Issue Type: Improvement
Reporter: Devaraj Das
Assignee: Devaraj Das
 Attachments: 5732-rpcengine-merge.11.patch, 
 5732-rpcengine-merge.12.patch, 5732-rpcengine-merge.7.patch, 
 rpcengine-merge.10.patch, rpcengine-merge.11.patch, rpcengine-merge.12.patch, 
 rpcengine-merge.3.patch, rpcengine-merge.4.patch, rpcengine-merge.9.patch, 
 rpcengine-merge.patch


 Remove the SecureRPCEngine and merge the security-related logic in the core 
 engine. Follow up to HBASE-5727.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5548) Add ability to get a table in the shell


[ 
https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13273037#comment-13273037
 ] 

Jesse Yates commented on HBASE-5548:


do we want to close this ticket now?

 Add ability to get a table in the shell
 ---

 Key: HBASE-5548
 URL: https://issues.apache.org/jira/browse/HBASE-5548
 Project: HBase
  Issue Type: Improvement
  Components: shell
Reporter: Jesse Yates
Assignee: Jesse Yates
 Fix For: 0.96.0

 Attachments: ruby_HBASE-5528-v0.patch, 
 ruby_HBASE-5548-addendum.patch, ruby_HBASE-5548-v1.patch, 
 ruby_HBASE-5548-v2.patch, ruby_HBASE-5548-v3.patch, ruby_HBASE-5548-v5.patch


 Currently, all the commands that operate on a table in the shell first have 
 to take the table as name as input. 
 There are two main considerations:
 * It is annoying to have to write the table name every time, when you should 
 just be able to get a reference to a table
 * the current implementation is very wasteful - it creates a new HTable for 
 each call (but reuses the connection since it uses the same configuration)
 We should be able to get a handle to a single HTable and then operate on that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4559) Refactor TestAvroServer into an integration test


 [ 
https://issues.apache.org/jira/browse/HBASE-4559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesse Yates updated HBASE-4559:
---

Resolution: Invalid
Status: Resolved  (was: Patch Available)

Marking invalid. With Keywals' work on the different sized tests, we don't need 
to worry about this.

 Refactor TestAvroServer into an integration test
 

 Key: HBASE-4559
 URL: https://issues.apache.org/jira/browse/HBASE-4559
 Project: HBase
  Issue Type: Improvement
  Components: test
Reporter: Jesse Yates
Assignee: Jesse Yates
 Attachments: java_HBASE_4559.txt


 TestAvroServer is a beefy test, spins up a mini cluster, does a large series 
 of manipulations and then spins it down. It take about 2 mins to run on a 
 local machine, which on the high side for a 'unit' test. 
 This is part of the implentation discussed in 
 http://search-hadoop.com/m/L9OzBNEOJK1

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5342) Grant/Revoke global permissions


[ 
https://issues.apache.org/jira/browse/HBASE-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13273040#comment-13273040
 ] 

Zhihong Yu commented on HBASE-5342:
---

For getTablePermissions():
{code}
 if (Bytes.equals(tableName, HConstants.ROOT_TABLE_NAME) ||
-Bytes.equals(tableName, HConstants.META_TABLE_NAME) ||
-Bytes.equals(tableName, AccessControlLists.ACL_TABLE_NAME)) {
+Bytes.equals(tableName, HConstants.META_TABLE_NAME)) {
{code}
Why is ACL_TABLE_NAME removed from the condition above ?

 Grant/Revoke global permissions
 ---

 Key: HBASE-5342
 URL: https://issues.apache.org/jira/browse/HBASE-5342
 Project: HBase
  Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Matteo Bertozzi
 Attachments: HBASE-5342-draft.patch, HBASE-5342-v0.patch, 
 HBASE-5342-v1.patch, HBASE-5342-v2.patch, HBASE-5342-v3.patch


 HBASE-3025 introduced simple ACLs based on coprocessors. It defines 
 global/table/cf/cq level permissions. However, there is no way to 
 grant/revoke global level permissions, other than the hbase.superuser conf 
 setting. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13273045#comment-13273045
]

Lars Hofhansl commented on HBASE-5986:
--

@Enis: yes HRegion.mutateRowsWithLock() has correct MVCC semantics even across
multi row scans.

The region boundary could be between start_key1 and mid_key1 if there rows
inserted and deleted before... Still unlikely. To be safe we could just have a
splitKeyPolicy that never splits the table prefix (i.e. all keys for the same
table are guaranteed to be in the region).

Clients can see holes in the META table when regions are being split

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode