[jira] [Updated] (HBASE-10005) TestVisibilityLabels fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-10005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-10005:
---
Affects Version/s: 0.98.0
Status: Patch Available (was: Open)
TestVisibilityLabels fails occasionally
---
Key: HBASE-10005
URL: https://issues.apache.org/jira/browse/HBASE-10005
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Ted Yu
Assignee: Anoop Sam John
Priority: Blocker
Fix For: 0.98.0
Attachments: HBASE-10005.patch,
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels-output.txt
I got the following test failures running test suite on hadoop-2 where
distributed log replay was turned on :
{code}
testAddVisibilityLabelsOnRSRestart(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.019 sec FAILURE!
java.lang.AssertionError: The count should be 8 expected:8 but was:6
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testAddVisibilityLabelsOnRSRestart(TestVisibilityLabels.java:408)
...
testClearUserAuths(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.002 sec FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testClearUserAuths(TestVisibilityLabels.java:505)
{code}
Logs to be attached
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-10005) TestVisibilityLabels fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-10005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-10005:
---
Attachment: HBASE-10005.patch
TestVisibilityLabels fails occasionally
---
Key: HBASE-10005
URL: https://issues.apache.org/jira/browse/HBASE-10005
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Ted Yu
Assignee: Anoop Sam John
Priority: Blocker
Fix For: 0.98.0
Attachments: HBASE-10005.patch,
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels-output.txt
I got the following test failures running test suite on hadoop-2 where
distributed log replay was turned on :
{code}
testAddVisibilityLabelsOnRSRestart(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.019 sec FAILURE!
java.lang.AssertionError: The count should be 8 expected:8 but was:6
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testAddVisibilityLabelsOnRSRestart(TestVisibilityLabels.java:408)
...
testClearUserAuths(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.002 sec FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testClearUserAuths(TestVisibilityLabels.java:505)
{code}
Logs to be attached
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10005) TestVisibilityLabels fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-10005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834589#comment-13834589
]
Anoop Sam John commented on HBASE-10005:
While doing the clearAuths we get the current Auths for the given user and
validate whether removing auth is already present for the user. This get
happens from the cache. The cache update happens via zk data changed event and
this thread can be bit late some times.
In test we 1st add auths and immediately clear some of them. When the
clearAuths reaches the cache update might not have happened yet!!
Changed the code so that it gets the current auths for the user from
hbase:labels table and not from cache. Any way these admin ops for the labels
will not happen frequently.
TestVisibilityLabels fails occasionally
---
Key: HBASE-10005
URL: https://issues.apache.org/jira/browse/HBASE-10005
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Ted Yu
Assignee: Anoop Sam John
Priority: Blocker
Fix For: 0.98.0
Attachments: HBASE-10005.patch,
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels-output.txt
I got the following test failures running test suite on hadoop-2 where
distributed log replay was turned on :
{code}
testAddVisibilityLabelsOnRSRestart(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.019 sec FAILURE!
java.lang.AssertionError: The count should be 8 expected:8 but was:6
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testAddVisibilityLabelsOnRSRestart(TestVisibilityLabels.java:408)
...
testClearUserAuths(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.002 sec FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testClearUserAuths(TestVisibilityLabels.java:505)
{code}
Logs to be attached
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10005) TestVisibilityLabels fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-10005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834599#comment-13834599
]
ramkrishna.s.vasudevan commented on HBASE-10005:
{code}
testcase time=0.011
classname=org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels
name=testClearUserAuths
failure type=java.lang.AssertionErrorjava.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testClearUserAuths(TestVisibilityLabels.java:505)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
{code}
Got this in the latest local test case run. One thing observed from the logs
is that there are no enough debug msgs.
{code}
2013-11-28 13:15:05,111 INFO [pool-1-thread-1] hbase.ResourceChecker(147):
before: security.visibility.TestVisibilityLabels#testClearUserAuths Thread=243,
OpenFileDescriptor=473, MaxFileDescriptor=327680, SystemLoadAverage=1204,
ProcessCount=367, AvailableMemoryMB=1914, ConnectionCount=3
2013-11-28 13:15:05,227 INFO [pool-1-thread-1] hbase.ResourceChecker(171):
after: security.visibility.TestVisibilityLabels#testClearUserAuths Thread=239
(was 243), OpenFileDescriptor=448 (was 473), MaxFileDescriptor=327680 (was
327680), SystemLoadAverage=1204 (was 1204), ProcessCount=367 (was 367),
AvailableMemoryMB=1861 (was 1914), ConnectionCount=3 (was 3)
2013-11-28 13:15:05,227 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(943):
Shutting down minicluster
{code}
We have only these two lines related to that testcase. May be the notification
of the zk should be the issue.
TestVisibilityLabels fails occasionally
---
Key: HBASE-10005
URL: https://issues.apache.org/jira/browse/HBASE-10005
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Ted Yu
Assignee: Anoop Sam John
Priority: Blocker
Fix For: 0.98.0
Attachments: HBASE-10005.patch,
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels-output.txt
I got the following test failures running test suite on hadoop-2 where
distributed log replay was turned on :
{code}
testAddVisibilityLabelsOnRSRestart(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.019 sec FAILURE!
java.lang.AssertionError: The count should be 8 expected:8 but was:6
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testAddVisibilityLabelsOnRSRestart(TestVisibilityLabels.java:408)
...
testClearUserAuths(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.002 sec FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testClearUserAuths(TestVisibilityLabels.java:505)
{code}
Logs to be attached
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10005) TestVisibilityLabels fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-10005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834601#comment-13834601
]
ramkrishna.s.vasudevan commented on HBASE-10005:
Okie same comment added by Anoop. Just saw that.
TestVisibilityLabels fails occasionally
---
Key: HBASE-10005
URL: https://issues.apache.org/jira/browse/HBASE-10005
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Ted Yu
Assignee: Anoop Sam John
Priority: Blocker
Fix For: 0.98.0
Attachments: HBASE-10005.patch,
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels-output.txt
I got the following test failures running test suite on hadoop-2 where
distributed log replay was turned on :
{code}
testAddVisibilityLabelsOnRSRestart(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.019 sec FAILURE!
java.lang.AssertionError: The count should be 8 expected:8 but was:6
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testAddVisibilityLabelsOnRSRestart(TestVisibilityLabels.java:408)
...
testClearUserAuths(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.002 sec FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testClearUserAuths(TestVisibilityLabels.java:505)
{code}
Logs to be attached
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-10049) Small improvments in region_mover.rb
[ https://issues.apache.org/jira/browse/HBASE-10049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-10049: Attachment: HBASE-10048-0.94-v2.diff [~yuzhih...@gmail.com] Updates: Close all tables at the end of region_mover.rb Small improvments in region_mover.rb Key: HBASE-10049 URL: https://issues.apache.org/jira/browse/HBASE-10049 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-10048-0.94-v2.diff, HBASE-10049-0.94-v1.diff We use region_mover.rb in the graceful upgrade of hbase cluster. Here are small improvements. a. remove the table.close(), because the htable could be reused. b. Add more info in the log of moving region. c. Add 20s sleep in load command to make sure the rs finished initialization of rpc server. There is a time gap between rs startup report and rpc server initialization. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8912) [0.94] AssignmentManager throws IllegalStateException from PENDING_OPEN to OFFLINE
[ https://issues.apache.org/jira/browse/HBASE-8912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834617#comment-13834617 ] chenjiajun commented on HBASE-8912: --- 2013-11-27 18:26:18,102 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: [] 2013-11-27 18:26:18,102 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state : H,http://istock.jrj.com.cn/article,002024,6567377.html,1385541132079.18c9cb11b3e673dec07038f166fb3ef7. state=PENDING_O PEN, ts=1385547978102, server=d199.uuc.com,60020,1385047501649 .. Cannot transit it to OFFLINE. java.lang.IllegalStateException: Unexpected state : H,http://istock.jrj.com.cn/article,002024,6567377.html,1385541132079.18c9cb11b3e673dec07038f166fb3ef7. state=PENDING_OPEN, ts=1385547978102, server=d199.uuc.com,60020,1385047501649 .. Cannot transit it to OFFLINE. at org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1890) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1690) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1426) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1398) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1393) at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2013-11-27 18:26:18,104 INFO org.apache.hadoop.hbase.master.HMaster: Aborting 2013-11-27 18:26:18,104 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 6 2013-11-27 18:26:18,104 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 6: exiting .. RS 's log: 2013-11-27 18:24:33,375 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://master.uc.uuc.com:9000/hbase/H/18c9cb11b3e673dec07038f166fb3ef7/.tmp/832ec249071c45b3934a186046ca429d: CompoundBloomFilterWriter 2013-11-27 18:24:33,385 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO DeleteFamily was added to HFile (hdfs://master.uc.uuc.com:9000/hbase/H/18c9cb11b3e673dec07038f166fb3ef7/.tmp/832ec249071c45b3934a186046ca429d) 2013-11-27 18:24:33,388 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at hdfs://master.uc.uuc.com:9000/hbase/H/18c9cb11b3e673dec07038f166fb3ef7/.tmp/832ec249071c45b3934a186046ca429d to hdfs://master.uc.uuc.com:9000/hbase/H/18c9cb11b3e673dec07038f166fb3ef7/page/832ec249071c45b3934a186046ca429d 2013-11-27 18:24:33,421 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 3 file(s) in page of H,http://istock.jrj.com.cn/article,002024,6567377.html,1385541132079.18c9cb11b3e673dec07038f166fb3ef7. into 832ec249071c45b3934a186046ca429d, size=85.7k; total size for store is 533.2m 2013-11-27 18:24:33,422 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=H,http://istock.jrj.com.cn/article,002024,6567377.html,1385541132079.18c9cb11b3e673dec07038f166fb3ef7., storeName=page, fileCount=3, fileSize=91.8k, priority=3, time=1230511587222436; duration=0sec 2013-11-27 18:24:37,189 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region DocSortByHour,\x00\x00\x011\xAE\xCB\xD8 \x00\x00,B\x00\x00\x05\x84\x00\x01\xE6\xAF\x00\x02\x09\x0D\x00\x00\x00\x00\x06\xB5c\xB3,1314093149893.c5e2ce4602fb62714c72cd8da3b50bd5. after a delay of 5137 2013-11-27 18:24:43,658 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultReplication 2013-11-27 18:24:43,658 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultBlockSize 2013-11-27 18:24:43,681 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://master.uc.uuc.com:9000/hbase/DocSortByHour/c5e2ce4602fb62714c72cd8da3b50bd5/.tmp/dd7c3908111d4fcc8945b2b6d5291364: CompoundBloomFilterWriter 2013-11-27 18:24:43,691 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO DeleteFamily was added to HFile (hdfs://master.uc.uuc.com:9000/hbase/DocSortByHour/c5e2ce4602fb62714c72cd8da3b50bd5/.tmp/dd7c3908111d4fcc8945b2b6d5291364) 2013-11-27 18:24:43,691 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=26736748922, memsize=832.0, into tmp file hdfs://master.uc.uuc.com:9000/hbase/DocSortByHour/c5e2ce4602fb62714c72cd8da3b50bd5/.tmp/dd7c3908111d4fcc8945b2b6d5291364 2013-11-27
[jira] [Commented] (HBASE-10050) graceful_stop.sh syntax error
[
https://issues.apache.org/jira/browse/HBASE-10050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834626#comment-13834626
]
Samir Ahmic commented on HBASE-10050:
-
I have tested script agains 0.94.6.1 cluster looks fine.
+1 for 0.94.
graceful_stop.sh syntax error
-
Key: HBASE-10050
URL: https://issues.apache.org/jira/browse/HBASE-10050
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.96.0
Environment: x86_64 GNU/Linux
Reporter: Samir Ahmic
Fix For: 0.98.0, 0.96.1
Attachments: HBASE-10050.patch
When i was trying to run graceful_stop.sh i got error:
{code}
./graceful_stop.sh: line 59: syntax error near unexpected token `;'
./graceful_stop.sh: line 59: `--failfast) ;'
{code}
After looking at script i notice that this lines are causing error:
{code}
--failfast) ;
-e) failfast=true; shift;;
--debug) ;
-d) debug=--debug; shift;;
{code}
They sholud be changed to this:
{code}
--failfast | -e) failfast=true; shift;;
--debug | -d) debug=--debug; shift;;
{code}
I will attach patch today.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-10049) Small improvments in region_mover.rb
[ https://issues.apache.org/jira/browse/HBASE-10049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-10049: Attachment: (was: HBASE-10048-0.94-v2.diff) Small improvments in region_mover.rb Key: HBASE-10049 URL: https://issues.apache.org/jira/browse/HBASE-10049 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-10049-0.94-v1.diff We use region_mover.rb in the graceful upgrade of hbase cluster. Here are small improvements. a. remove the table.close(), because the htable could be reused. b. Add more info in the log of moving region. c. Add 20s sleep in load command to make sure the rs finished initialization of rpc server. There is a time gap between rs startup report and rpc server initialization. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-10049) Small improvments in region_mover.rb
[ https://issues.apache.org/jira/browse/HBASE-10049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-10049: Attachment: HBASE-10049-0.94-v2.diff Patch for 0.94 Small improvments in region_mover.rb Key: HBASE-10049 URL: https://issues.apache.org/jira/browse/HBASE-10049 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-10049-0.94-v1.diff, HBASE-10049-0.94-v2.diff We use region_mover.rb in the graceful upgrade of hbase cluster. Here are small improvements. a. remove the table.close(), because the htable could be reused. b. Add more info in the log of moving region. c. Add 20s sleep in load command to make sure the rs finished initialization of rpc server. There is a time gap between rs startup report and rpc server initialization. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-10049) Small improvments in region_mover.rb
[ https://issues.apache.org/jira/browse/HBASE-10049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-10049: Attachment: HBASE-10049-trunk-v1.diff Patch for trunk Small improvments in region_mover.rb Key: HBASE-10049 URL: https://issues.apache.org/jira/browse/HBASE-10049 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-10049-0.94-v1.diff, HBASE-10049-0.94-v2.diff, HBASE-10049-trunk-v1.diff We use region_mover.rb in the graceful upgrade of hbase cluster. Here are small improvements. a. remove the table.close(), because the htable could be reused. b. Add more info in the log of moving region. c. Add 20s sleep in load command to make sure the rs finished initialization of rpc server. There is a time gap between rs startup report and rpc server initialization. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-10049) Small improvments in region_mover.rb
[ https://issues.apache.org/jira/browse/HBASE-10049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-10049: --- Status: Patch Available (was: Open) Small improvments in region_mover.rb Key: HBASE-10049 URL: https://issues.apache.org/jira/browse/HBASE-10049 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-10049-0.94-v1.diff, HBASE-10049-0.94-v2.diff, HBASE-10049-trunk-v1.diff We use region_mover.rb in the graceful upgrade of hbase cluster. Here are small improvements. a. remove the table.close(), because the htable could be reused. b. Add more info in the log of moving region. c. Add 20s sleep in load command to make sure the rs finished initialization of rpc server. There is a time gap between rs startup report and rpc server initialization. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10048) Add hlog number metric in regionserver
[ https://issues.apache.org/jira/browse/HBASE-10048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834638#comment-13834638 ] Ted Yu commented on HBASE-10048: Trunk patch please. Add hlog number metric in regionserver -- Key: HBASE-10048 URL: https://issues.apache.org/jira/browse/HBASE-10048 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-10048-0.94-v1.diff, HBASE-10048-0.94-v2.diff Add hlog number metric in regionserver. We can use this metric to alert about memstore flush because of too many hlogs. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10046) Unmonitored HBase service could accumulate Status objects and OOM
[
https://issues.apache.org/jira/browse/HBASE-10046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834640#comment-13834640
]
Hudson commented on HBASE-10046:
SUCCESS: Integrated in hbase-0.96 #206 (See
[https://builds.apache.org/job/hbase-0.96/206/])
HBASE-10046 Unmonitored HBase service could accumulate Status objects and OOM
(tedyu: rev 1546095)
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/TaskMonitor.java
Unmonitored HBase service could accumulate Status objects and OOM
-
Key: HBASE-10046
URL: https://issues.apache.org/jira/browse/HBASE-10046
Project: HBase
Issue Type: Bug
Components: monitoring
Affects Versions: 0.96.0, 0.94.14
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Fix For: 0.98.0, 0.96.1, 0.94.15
Attachments: HBASE-10046.patch, HBASE-10046_0.94.patch
This was observed in a cluster where HBase Master UI was not monitored for
quite a while. During this period, a distributed log spitting task failed in
an in-completable way and Master kept recreating the task over and over again.
And each such task would create a monitored status...
{code:title=SplitLogManager.java}
...
public long splitLogDistributed(final ListPath logDirs, PathFilter
filter)
throws IOException {
MonitoredTask status = TaskMonitor.get().createStatus(
Doing distributed log split in + logDirs);
...
{code}
...which kept accumulating on heap.
Now these monitored tasks get cleaned only when someone looks at the service
UI otherwise they keep growing boundless.
The postmortem of the heap dump showed that these task object occupied 99% of
HBase master heap accumulated over a week.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10020) Add maven compile-protobuf profile
[
https://issues.apache.org/jira/browse/HBASE-10020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834641#comment-13834641
]
Hudson commented on HBASE-10020:
SUCCESS: Integrated in hbase-0.96 #206 (See
[https://builds.apache.org/job/hbase-0.96/206/])
HBASE-10020 Add maven compile-protobuf profile (enis: rev 1546241)
* /hbase/branches/0.96/dev-support/build-proto.sh
* /hbase/branches/0.96/hbase-examples/pom.xml
*
/hbase/branches/0.96/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/generated/BulkDeleteProtos.java
*
/hbase/branches/0.96/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/generated/ExampleProtos.java
* /hbase/branches/0.96/hbase-protocol/README.txt
* /hbase/branches/0.96/hbase-protocol/pom.xml
* /hbase/branches/0.96/hbase-server/pom.xml
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/ColumnSchemaMessage.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/ScannerMessage.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/StorageClusterStatusMessage.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/TableInfoMessage.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/TableSchemaMessage.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/VersionMessage.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/protobuf/generated/PingProtos.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/protobuf/generated/TestProtos.java
* /hbase/branches/0.96/pom.xml
* /hbase/branches/0.96/src/main/docbkx/developer.xml
Add maven compile-protobuf profile
--
Key: HBASE-10020
URL: https://issues.apache.org/jira/browse/HBASE-10020
Project: HBase
Issue Type: Improvement
Components: build
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.0, 0.96.1
Attachments: hbase-10020_v1.patch, hbase-10020_v2.patch
Ad a maven profile to compile protobufs instead of the dev-support script
which is only for hbase-protocol module.
{code}
mvn test-compile -Dcompile-protobuf
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10051) rolling-restart.sh have old value of zookeeper.znode.unassiged config causing infinite loop
[
https://issues.apache.org/jira/browse/HBASE-10051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834643#comment-13834643
]
Hudson commented on HBASE-10051:
SUCCESS: Integrated in hbase-0.96 #206 (See
[https://builds.apache.org/job/hbase-0.96/206/])
HBASE-10051 rolling-restart.sh have old value of zookeeper.znode.unassiged
config causing infinite loop (stack: rev 1546214)
* /hbase/branches/0.96/bin/rolling-restart.sh
rolling-restart.sh have old value of zookeeper.znode.unassiged config
causing infinite loop
-
Key: HBASE-10051
URL: https://issues.apache.org/jira/browse/HBASE-10051
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.96.0
Environment: x86_64 GNU/Linux
Reporter: Samir Ahmic
Assignee: Samir Ahmic
Fix For: 0.98.0, 0.96.1
Attachments: HBASE-10051.patch
While running rolling-restart.sh without arguments or with --master-only
argument script throws error when trying to check did all regions are cleared
from transition state. Here is line causing error:
{code}
if [ $zunassigned == null ]; then zunassigned=unassigned; fi
{code}
AFAIK default value of zookeeper.znode.unassiged is changed to
region-in-transition. So we should change this line to:
{code}
if [ $zunassigned == null ]; then zunassigned=region-in-transition; fi
{code}
I will attach patch today.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10053) ProtobufUtil#toMutationNoData interface change causes downstream project build break
[
https://issues.apache.org/jira/browse/HBASE-10053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834644#comment-13834644
]
Hudson commented on HBASE-10053:
SUCCESS: Integrated in hbase-0.96 #206 (See
[https://builds.apache.org/job/hbase-0.96/206/])
hbase-10053: ProtobufUtil#toMutationNoData interface change causes downstream
project build break (jeffreyz: rev 1546249)
*
/hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
ProtobufUtil#toMutationNoData interface change causes downstream project
build break
Key: HBASE-10053
URL: https://issues.apache.org/jira/browse/HBASE-10053
Project: HBase
Issue Type: Bug
Reporter: Jeffrey Zhong
Assignee: Jeffrey Zhong
Priority: Critical
Fix For: 0.98.0, 0.96.1
Attachments: hbase-10053.patch
{code}
public static MutationProto toMutationNoData(final MutationType type, final
Mutation mutation)
{code}
is gone which causes Hive build break.
{code}
[javac]
.../hbase-handler/src/java/org/apache/hadoop/hive/hbase/PutWritable.java:65:
toMutationNoData(org.apache.hadoop.hbase.protobuf.generated.ClientProtos.MutationProto.MutationType,org.apache.hadoop.hbase.client.Mutation,org.apache.hadoop.hbase.protobuf.generated.ClientProtos.MutationProto.Builder)
in org.apache.hadoop.hbase.protobuf.ProtobufUtil cannot be applied to
(org.apache.hadoop.hbase.protobuf.generated.ClientProtos.MutationProto.MutationType,org.apache.hadoop.hbase.client.Put)
[javac] ProtobufUtil.toMutationNoData(MutationType.PUT,
put).writeDelimitedTo(DataOutputOutputStream.from(out));
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10050) graceful_stop.sh syntax error
[
https://issues.apache.org/jira/browse/HBASE-10050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834642#comment-13834642
]
Hudson commented on HBASE-10050:
SUCCESS: Integrated in hbase-0.96 #206 (See
[https://builds.apache.org/job/hbase-0.96/206/])
HBASE-10050 graceful_stop.sh syntax error (stack: rev 1546213)
* /hbase/branches/0.96/bin/graceful_stop.sh
graceful_stop.sh syntax error
-
Key: HBASE-10050
URL: https://issues.apache.org/jira/browse/HBASE-10050
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.96.0
Environment: x86_64 GNU/Linux
Reporter: Samir Ahmic
Fix For: 0.98.0, 0.96.1
Attachments: HBASE-10050.patch
When i was trying to run graceful_stop.sh i got error:
{code}
./graceful_stop.sh: line 59: syntax error near unexpected token `;'
./graceful_stop.sh: line 59: `--failfast) ;'
{code}
After looking at script i notice that this lines are causing error:
{code}
--failfast) ;
-e) failfast=true; shift;;
--debug) ;
-d) debug=--debug; shift;;
{code}
They sholud be changed to this:
{code}
--failfast | -e) failfast=true; shift;;
--debug | -d) debug=--debug; shift;;
{code}
I will attach patch today.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10046) Unmonitored HBase service could accumulate Status objects and OOM
[
https://issues.apache.org/jira/browse/HBASE-10046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834649#comment-13834649
]
Hudson commented on HBASE-10046:
FAILURE: Integrated in HBase-0.94 #1213 (See
[https://builds.apache.org/job/HBase-0.94/1213/])
HBASE-10046 Unmonitored HBase service could accumulate Status objects and OOM
(tedyu: rev 1546098)
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/monitoring/TaskMonitor.java
Unmonitored HBase service could accumulate Status objects and OOM
-
Key: HBASE-10046
URL: https://issues.apache.org/jira/browse/HBASE-10046
Project: HBase
Issue Type: Bug
Components: monitoring
Affects Versions: 0.96.0, 0.94.14
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Fix For: 0.98.0, 0.96.1, 0.94.15
Attachments: HBASE-10046.patch, HBASE-10046_0.94.patch
This was observed in a cluster where HBase Master UI was not monitored for
quite a while. During this period, a distributed log spitting task failed in
an in-completable way and Master kept recreating the task over and over again.
And each such task would create a monitored status...
{code:title=SplitLogManager.java}
...
public long splitLogDistributed(final ListPath logDirs, PathFilter
filter)
throws IOException {
MonitoredTask status = TaskMonitor.get().createStatus(
Doing distributed log split in + logDirs);
...
{code}
...which kept accumulating on heap.
Now these monitored tasks get cleaned only when someone looks at the service
UI otherwise they keep growing boundless.
The postmortem of the heap dump showed that these task object occupied 99% of
HBase master heap accumulated over a week.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9832) Add MR support for Visibility labels
[ https://issues.apache.org/jira/browse/HBASE-9832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834658#comment-13834658 ] Anoop Sam John commented on HBASE-9832: --- Rather than taking the visibility expression in the file as attributes, can we add a new column for visibility? Like RK column and TS column. Fix a name for the column and that data will be used as ColumnVisibility. When it is TableOutputFormat (Map alone job) set this visibility to Put CellVisibility. When it is HFileOutputFormat, use the expression and convert to tag and set in every KV. Add MR support for Visibility labels Key: HBASE-9832 URL: https://issues.apache.org/jira/browse/HBASE-9832 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.98.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 Attachments: HBASE-9832.patch, HBASE-9832_1.patch MR needs to support adding the visibility labels through TableOutputFormat and HfileOutPutformat. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10005) TestVisibilityLabels fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-10005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834661#comment-13834661
]
Anoop Sam John commented on HBASE-10005:
Yes I was able to reproduce the issue with break points. Yes I also felt we
missed some debug logs. Let me check other areas also and add more debug logs.
Will add it in another IA jira. You +1 for this patch Ram? Let us see what QA
says
TestVisibilityLabels fails occasionally
---
Key: HBASE-10005
URL: https://issues.apache.org/jira/browse/HBASE-10005
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Ted Yu
Assignee: Anoop Sam John
Priority: Blocker
Fix For: 0.98.0
Attachments: HBASE-10005.patch,
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels-output.txt
I got the following test failures running test suite on hadoop-2 where
distributed log replay was turned on :
{code}
testAddVisibilityLabelsOnRSRestart(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.019 sec FAILURE!
java.lang.AssertionError: The count should be 8 expected:8 but was:6
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testAddVisibilityLabelsOnRSRestart(TestVisibilityLabels.java:408)
...
testClearUserAuths(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.002 sec FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testClearUserAuths(TestVisibilityLabels.java:505)
{code}
Logs to be attached
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10005) TestVisibilityLabels fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-10005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834664#comment-13834664
]
Hadoop QA commented on HBASE-10005:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616206/HBASE-10005.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8020//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8020//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8020//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8020//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8020//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8020//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8020//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8020//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8020//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8020//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8020//console
This message is automatically generated.
TestVisibilityLabels fails occasionally
---
Key: HBASE-10005
URL: https://issues.apache.org/jira/browse/HBASE-10005
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Ted Yu
Assignee: Anoop Sam John
Priority: Blocker
Fix For: 0.98.0
Attachments: HBASE-10005.patch,
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels-output.txt
I got the following test failures running test suite on hadoop-2 where
distributed log replay was turned on :
{code}
testAddVisibilityLabelsOnRSRestart(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.019 sec FAILURE!
java.lang.AssertionError: The count should be 8 expected:8 but was:6
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testAddVisibilityLabelsOnRSRestart(TestVisibilityLabels.java:408)
...
testClearUserAuths(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.002 sec FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testClearUserAuths(TestVisibilityLabels.java:505)
{code}
Logs to be attached
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10050) graceful_stop.sh syntax error
[
https://issues.apache.org/jira/browse/HBASE-10050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834672#comment-13834672
]
Hudson commented on HBASE-10050:
SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #854 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/854/])
HBASE-10050 graceful_stop.sh syntax error (stack: rev 1546212)
* /hbase/trunk/bin/graceful_stop.sh
graceful_stop.sh syntax error
-
Key: HBASE-10050
URL: https://issues.apache.org/jira/browse/HBASE-10050
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.96.0
Environment: x86_64 GNU/Linux
Reporter: Samir Ahmic
Fix For: 0.98.0, 0.96.1
Attachments: HBASE-10050.patch
When i was trying to run graceful_stop.sh i got error:
{code}
./graceful_stop.sh: line 59: syntax error near unexpected token `;'
./graceful_stop.sh: line 59: `--failfast) ;'
{code}
After looking at script i notice that this lines are causing error:
{code}
--failfast) ;
-e) failfast=true; shift;;
--debug) ;
-d) debug=--debug; shift;;
{code}
They sholud be changed to this:
{code}
--failfast | -e) failfast=true; shift;;
--debug | -d) debug=--debug; shift;;
{code}
I will attach patch today.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-7662) [Per-KV security] Per cell ACLs stored in tags
[ https://issues.apache.org/jira/browse/HBASE-7662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834669#comment-13834669 ] Hudson commented on HBASE-7662: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #854 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/854/]) HBASE-7662 [Per-KV security] Per cell ACLs stored in tags - Addendum to fix javadoc warnings (anoopsamjohn: rev 1546159) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Query.java [Per-KV security] Per cell ACLs stored in tags -- Key: HBASE-7662 URL: https://issues.apache.org/jira/browse/HBASE-7662 Project: HBase Issue Type: Sub-task Components: Coprocessors, security Affects Versions: 0.98.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.98.0 Attachments: 7662-final.patch, 7662-final.patch, 7662.patch, 7662.patch, 7662.patch, 7662.patch, HBASE-7662_addendum.patch, latency-single.7662.xlsx We can improve the performance of per-cell authorization if the read of the cell ACL, if any, is combined with the sequential read of the cell data already in progress. When tags are inlined with KVs in block encoding (see HBASE-7448, and more generally HBASE-7233), we can use them to carry cell ACLs instead of using out-of-line storage (HBASE-7661) for that purpose. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10046) Unmonitored HBase service could accumulate Status objects and OOM
[
https://issues.apache.org/jira/browse/HBASE-10046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834670#comment-13834670
]
Hudson commented on HBASE-10046:
SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #854 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/854/])
HBASE-10046 Unmonitored HBase service could accumulate Status objects and OOM
(tedyu: rev 1546094)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/TaskMonitor.java
Unmonitored HBase service could accumulate Status objects and OOM
-
Key: HBASE-10046
URL: https://issues.apache.org/jira/browse/HBASE-10046
Project: HBase
Issue Type: Bug
Components: monitoring
Affects Versions: 0.96.0, 0.94.14
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Fix For: 0.98.0, 0.96.1, 0.94.15
Attachments: HBASE-10046.patch, HBASE-10046_0.94.patch
This was observed in a cluster where HBase Master UI was not monitored for
quite a while. During this period, a distributed log spitting task failed in
an in-completable way and Master kept recreating the task over and over again.
And each such task would create a monitored status...
{code:title=SplitLogManager.java}
...
public long splitLogDistributed(final ListPath logDirs, PathFilter
filter)
throws IOException {
MonitoredTask status = TaskMonitor.get().createStatus(
Doing distributed log split in + logDirs);
...
{code}
...which kept accumulating on heap.
Now these monitored tasks get cleaned only when someone looks at the service
UI otherwise they keep growing boundless.
The postmortem of the heap dump showed that these task object occupied 99% of
HBase master heap accumulated over a week.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10051) rolling-restart.sh have old value of zookeeper.znode.unassiged config causing infinite loop
[
https://issues.apache.org/jira/browse/HBASE-10051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834673#comment-13834673
]
Hudson commented on HBASE-10051:
SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #854 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/854/])
HBASE-10051 rolling-restart.sh have old value of zookeeper.znode.unassiged
config causing infinite loop (stack: rev 1546215)
* /hbase/trunk/bin/rolling-restart.sh
rolling-restart.sh have old value of zookeeper.znode.unassiged config
causing infinite loop
-
Key: HBASE-10051
URL: https://issues.apache.org/jira/browse/HBASE-10051
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.96.0
Environment: x86_64 GNU/Linux
Reporter: Samir Ahmic
Assignee: Samir Ahmic
Fix For: 0.98.0, 0.96.1
Attachments: HBASE-10051.patch
While running rolling-restart.sh without arguments or with --master-only
argument script throws error when trying to check did all regions are cleared
from transition state. Here is line causing error:
{code}
if [ $zunassigned == null ]; then zunassigned=unassigned; fi
{code}
AFAIK default value of zookeeper.znode.unassiged is changed to
region-in-transition. So we should change this line to:
{code}
if [ $zunassigned == null ]; then zunassigned=region-in-transition; fi
{code}
I will attach patch today.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10020) Add maven compile-protobuf profile
[
https://issues.apache.org/jira/browse/HBASE-10020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834671#comment-13834671
]
Hudson commented on HBASE-10020:
SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #854 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/854/])
HBASE-10020 Add maven compile-protobuf profile (enis: rev 1546237)
* /hbase/trunk/dev-support/build-proto.sh
* /hbase/trunk/hbase-examples/pom.xml
*
/hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/generated/BulkDeleteProtos.java
*
/hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/generated/ExampleProtos.java
* /hbase/trunk/hbase-protocol/README.txt
* /hbase/trunk/hbase-protocol/pom.xml
* /hbase/trunk/hbase-server/pom.xml
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/ColumnSchemaMessage.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/ScannerMessage.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/StorageClusterStatusMessage.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/TableInfoMessage.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/TableSchemaMessage.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/VersionMessage.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/protobuf/generated/PingProtos.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/protobuf/generated/TestProtos.java
* /hbase/trunk/pom.xml
* /hbase/trunk/src/main/docbkx/developer.xml
Add maven compile-protobuf profile
--
Key: HBASE-10020
URL: https://issues.apache.org/jira/browse/HBASE-10020
Project: HBase
Issue Type: Improvement
Components: build
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.0, 0.96.1
Attachments: hbase-10020_v1.patch, hbase-10020_v2.patch
Ad a maven profile to compile protobufs instead of the dev-support script
which is only for hbase-protocol module.
{code}
mvn test-compile -Dcompile-protobuf
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10053) ProtobufUtil#toMutationNoData interface change causes downstream project build break
[
https://issues.apache.org/jira/browse/HBASE-10053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834674#comment-13834674
]
Hudson commented on HBASE-10053:
SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #854 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/854/])
hbase-10053: ProtobufUtil#toMutationNoData interface change causes downstream
project build break (jeffreyz: rev 1546248)
*
/hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
ProtobufUtil#toMutationNoData interface change causes downstream project
build break
Key: HBASE-10053
URL: https://issues.apache.org/jira/browse/HBASE-10053
Project: HBase
Issue Type: Bug
Reporter: Jeffrey Zhong
Assignee: Jeffrey Zhong
Priority: Critical
Fix For: 0.98.0, 0.96.1
Attachments: hbase-10053.patch
{code}
public static MutationProto toMutationNoData(final MutationType type, final
Mutation mutation)
{code}
is gone which causes Hive build break.
{code}
[javac]
.../hbase-handler/src/java/org/apache/hadoop/hive/hbase/PutWritable.java:65:
toMutationNoData(org.apache.hadoop.hbase.protobuf.generated.ClientProtos.MutationProto.MutationType,org.apache.hadoop.hbase.client.Mutation,org.apache.hadoop.hbase.protobuf.generated.ClientProtos.MutationProto.Builder)
in org.apache.hadoop.hbase.protobuf.ProtobufUtil cannot be applied to
(org.apache.hadoop.hbase.protobuf.generated.ClientProtos.MutationProto.MutationType,org.apache.hadoop.hbase.client.Put)
[javac] ProtobufUtil.toMutationNoData(MutationType.PUT,
put).writeDelimitedTo(DataOutputOutputStream.from(out));
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10005) TestVisibilityLabels fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-10005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834684#comment-13834684
]
ramkrishna.s.vasudevan commented on HBASE-10005:
Yes +1 for now. We can see if this happens again. May be after commit we can
have this JIRA open for couple of days. If nothing fails close this. +1 for
adding some more debug logs.
I have not checked this failure till now testAddVisibilityLabelsOnRSRestart.
Does this also repeat?
TestVisibilityLabels fails occasionally
---
Key: HBASE-10005
URL: https://issues.apache.org/jira/browse/HBASE-10005
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Ted Yu
Assignee: Anoop Sam John
Priority: Blocker
Fix For: 0.98.0
Attachments: HBASE-10005.patch,
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels-output.txt
I got the following test failures running test suite on hadoop-2 where
distributed log replay was turned on :
{code}
testAddVisibilityLabelsOnRSRestart(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.019 sec FAILURE!
java.lang.AssertionError: The count should be 8 expected:8 but was:6
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testAddVisibilityLabelsOnRSRestart(TestVisibilityLabels.java:408)
...
testClearUserAuths(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.002 sec FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testClearUserAuths(TestVisibilityLabels.java:505)
{code}
Logs to be attached
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9832) Add MR support for Visibility labels
[ https://issues.apache.org/jira/browse/HBASE-9832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834687#comment-13834687 ] ramkrishna.s.vasudevan commented on HBASE-9832: --- So we will add everything like ACLs also as new columns only and then use them in the core code itself? Add MR support for Visibility labels Key: HBASE-9832 URL: https://issues.apache.org/jira/browse/HBASE-9832 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.98.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 Attachments: HBASE-9832.patch, HBASE-9832_1.patch MR needs to support adding the visibility labels through TableOutputFormat and HfileOutPutformat. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9884) Add Thrift and REST support for Visibility Labels
[ https://issues.apache.org/jira/browse/HBASE-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-9884: -- Status: Open (was: Patch Available) Add Thrift and REST support for Visibility Labels - Key: HBASE-9884 URL: https://issues.apache.org/jira/browse/HBASE-9884 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.98.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 Attachments: HBASE-9884.patch In HBASE-7663 the REST and thrift support has been seperated out because the patch is becoming bigger. This JIRA is to add the Thrift and REST part as a seperated patch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9884) Add Thrift and REST support for Visibility Labels
[ https://issues.apache.org/jira/browse/HBASE-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-9884: -- Attachment: HBASE-9884_1.patch Updated patch. Testcases passes with this. Add Thrift and REST support for Visibility Labels - Key: HBASE-9884 URL: https://issues.apache.org/jira/browse/HBASE-9884 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.98.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 Attachments: HBASE-9884.patch, HBASE-9884_1.patch In HBASE-7663 the REST and thrift support has been seperated out because the patch is becoming bigger. This JIRA is to add the Thrift and REST part as a seperated patch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9884) Add Thrift and REST support for Visibility Labels
[ https://issues.apache.org/jira/browse/HBASE-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-9884: -- Status: Patch Available (was: Open) Add Thrift and REST support for Visibility Labels - Key: HBASE-9884 URL: https://issues.apache.org/jira/browse/HBASE-9884 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.98.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 Attachments: HBASE-9884.patch, HBASE-9884_1.patch In HBASE-7663 the REST and thrift support has been seperated out because the patch is becoming bigger. This JIRA is to add the Thrift and REST part as a seperated patch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9884) Add Thrift and REST support for Visibility Labels
[
https://issues.apache.org/jira/browse/HBASE-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834691#comment-13834691
]
Hadoop QA commented on HBASE-9884:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616225/HBASE-9884_1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8022//console
This message is automatically generated.
Add Thrift and REST support for Visibility Labels
-
Key: HBASE-9884
URL: https://issues.apache.org/jira/browse/HBASE-9884
Project: HBase
Issue Type: Improvement
Components: Client
Affects Versions: 0.98.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.98.0
Attachments: HBASE-9884.patch, HBASE-9884_1.patch
In HBASE-7663 the REST and thrift support has been seperated out because the
patch is becoming bigger. This JIRA is to add the Thrift and REST part as a
seperated patch.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834693#comment-13834693 ] Liang Xie commented on HBASE-9535: -- [~nkeywal] It would be great to add some mem used counters arond ElasticByteBufferPool, that will be useful while we encourntered a OOM Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack Attachments: 9535.v1.patch, 9535.v2.patch ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9884) Add Thrift and REST support for Visibility Labels
[ https://issues.apache.org/jira/browse/HBASE-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-9884: -- Status: Open (was: Patch Available) Add Thrift and REST support for Visibility Labels - Key: HBASE-9884 URL: https://issues.apache.org/jira/browse/HBASE-9884 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.98.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 Attachments: HBASE-9884.patch, HBASE-9884_1.patch In HBASE-7663 the REST and thrift support has been seperated out because the patch is becoming bigger. This JIRA is to add the Thrift and REST part as a seperated patch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-4811) Support reverse Scan
[
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834706#comment-13834706
]
Ted Yu commented on HBASE-4811:
---
+1 from me
Support reverse Scan
Key: HBASE-4811
URL: https://issues.apache.org/jira/browse/HBASE-4811
Project: HBase
Issue Type: New Feature
Components: Client
Affects Versions: 0.20.6, 0.94.7
Reporter: John Carrino
Assignee: chunhui shen
Fix For: 0.98.0, 0.96.1, 0.94.15
Attachments: 4811-0.94-v22.txt, 4811-0.94-v23.txt, 4811-0.94-v3.txt,
4811-trunk-v10.txt, 4811-trunk-v5.patch, HBase-4811-0.94-v2.txt,
HBase-4811-0.94.3modified.txt, hbase-4811-0.94 v21.patch,
hbase-4811-0.94-v24.patch, hbase-4811-trunkv1.patch,
hbase-4811-trunkv11.patch, hbase-4811-trunkv12.patch,
hbase-4811-trunkv13.patch, hbase-4811-trunkv14.patch,
hbase-4811-trunkv15.patch, hbase-4811-trunkv16.patch,
hbase-4811-trunkv17.patch, hbase-4811-trunkv18.patch,
hbase-4811-trunkv19.patch, hbase-4811-trunkv20.patch,
hbase-4811-trunkv21.patch, hbase-4811-trunkv24.patch,
hbase-4811-trunkv24.patch, hbase-4811-trunkv25.patch,
hbase-4811-trunkv26.patch, hbase-4811-trunkv27.patch,
hbase-4811-trunkv28.patch, hbase-4811-trunkv4.patch,
hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch,
hbase-4811-trunkv9.patch
Reversed scan means scan the rows backward.
And StartRow bigger than StopRow in a reversed scan.
For example, for the following rows:
aaa/c1:q1/value1
aaa/c1:q2/value2
bbb/c1:q1/value1
bbb/c1:q2/value2
ccc/c1:q1/value1
ccc/c1:q2/value2
ddd/c1:q1/value1
ddd/c1:q2/value2
eee/c1:q1/value1
eee/c1:q2/value2
you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
Scan scan = new Scan();
scan.setStartRow('ddd');
scan.setStopRow('bbb');
scan.setReversed(true);
for(Result result:htable.getScanner(scan)){
System.out.println(result);
}
Aslo you could do the reversed scan with shell like this:
hbase scan 'table',{REVERSED = true,STARTROW='ddd', STOPROW='bbb'}
And the output is:
ddd/c1:q1/value1
ddd/c1:q2/value2
ccc/c1:q1/value1
ccc/c1:q2/value2
NOTE: when setting reversed as true for a client scan, you must set the start
row, else will throw exception. Through {@link
Scan#createBiggestByteArray(int)},you could get a big enough byte array as
the start row
All the documentation I find about HBase says that if you want forward and
reverse scans you should just build 2 tables and one be ascending and one
descending. Is there a fundamental reason that HBase only supports forward
Scan? It seems like a lot of extra space overhead and coding overhead (to
keep them in sync) to support 2 tables.
I am assuming this has been discussed before, but I can't find the
discussions anywhere about it or why it would be infeasible.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834707#comment-13834707 ] Nicolas Liochon commented on HBASE-9535: bq. It would be great to add some mem used counters arond ElasticByteBufferPool, that will be useful while we encourntered a OOM Yeah, I agree. I'm actually wondering if I have to integrate it to the metrics from day one or if I can wait. Thanks for looking at the patch. It's still early stage, yes. If you can try it from a performance point of view, I'm interested. I would like to have a good enough (i.e. fully implemented with the right configuration parameters available) version for tomorrow. Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack Attachments: 9535.v1.patch, 9535.v2.patch ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9884) Add Thrift and REST support for Visibility Labels
[ https://issues.apache.org/jira/browse/HBASE-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-9884: -- Status: Patch Available (was: Open) Add Thrift and REST support for Visibility Labels - Key: HBASE-9884 URL: https://issues.apache.org/jira/browse/HBASE-9884 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.98.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 Attachments: HBASE-9884.patch, HBASE-9884_1.patch, HBASE-9884_2.patch In HBASE-7663 the REST and thrift support has been seperated out because the patch is becoming bigger. This JIRA is to add the Thrift and REST part as a seperated patch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9884) Add Thrift and REST support for Visibility Labels
[ https://issues.apache.org/jira/browse/HBASE-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-9884: -- Attachment: HBASE-9884_2.patch Some update happened by the time i added the previous patch. Rebased again. Add Thrift and REST support for Visibility Labels - Key: HBASE-9884 URL: https://issues.apache.org/jira/browse/HBASE-9884 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.98.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 Attachments: HBASE-9884.patch, HBASE-9884_1.patch, HBASE-9884_2.patch In HBASE-7663 the REST and thrift support has been seperated out because the patch is becoming bigger. This JIRA is to add the Thrift and REST part as a seperated patch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10049) Small improvments in region_mover.rb
[
https://issues.apache.org/jira/browse/HBASE-10049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834732#comment-13834732
]
Hadoop QA commented on HBASE-10049:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616215/HBASE-10049-trunk-v1.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8021//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8021//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8021//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8021//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8021//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8021//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8021//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8021//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8021//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8021//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8021//console
This message is automatically generated.
Small improvments in region_mover.rb
Key: HBASE-10049
URL: https://issues.apache.org/jira/browse/HBASE-10049
Project: HBase
Issue Type: Improvement
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
Attachments: HBASE-10049-0.94-v1.diff, HBASE-10049-0.94-v2.diff,
HBASE-10049-trunk-v1.diff
We use region_mover.rb in the graceful upgrade of hbase cluster.
Here are small improvements.
a. remove the table.close(), because the htable could be reused.
b. Add more info in the log of moving region.
c. Add 20s sleep in load command to make sure the rs finished initialization
of rpc server. There is a time gap between rs startup report and rpc server
initialization.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-10048) Add hlog number metric in regionserver
[ https://issues.apache.org/jira/browse/HBASE-10048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-10048: Attachment: HBASE-10048-trunk-v1.diff Patch for trunk. Don't +1 in getNumLogFiles, for it is used in many others places. Add hlog number metric in regionserver -- Key: HBASE-10048 URL: https://issues.apache.org/jira/browse/HBASE-10048 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-10048-0.94-v1.diff, HBASE-10048-0.94-v2.diff, HBASE-10048-trunk-v1.diff Add hlog number metric in regionserver. We can use this metric to alert about memstore flush because of too many hlogs. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10046) Unmonitored HBase service could accumulate Status objects and OOM
[
https://issues.apache.org/jira/browse/HBASE-10046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834744#comment-13834744
]
Hudson commented on HBASE-10046:
FAILURE: Integrated in hbase-0.96-hadoop2 #134 (See
[https://builds.apache.org/job/hbase-0.96-hadoop2/134/])
HBASE-10046 Unmonitored HBase service could accumulate Status objects and OOM
(tedyu: rev 1546095)
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/TaskMonitor.java
Unmonitored HBase service could accumulate Status objects and OOM
-
Key: HBASE-10046
URL: https://issues.apache.org/jira/browse/HBASE-10046
Project: HBase
Issue Type: Bug
Components: monitoring
Affects Versions: 0.96.0, 0.94.14
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Fix For: 0.98.0, 0.96.1, 0.94.15
Attachments: HBASE-10046.patch, HBASE-10046_0.94.patch
This was observed in a cluster where HBase Master UI was not monitored for
quite a while. During this period, a distributed log spitting task failed in
an in-completable way and Master kept recreating the task over and over again.
And each such task would create a monitored status...
{code:title=SplitLogManager.java}
...
public long splitLogDistributed(final ListPath logDirs, PathFilter
filter)
throws IOException {
MonitoredTask status = TaskMonitor.get().createStatus(
Doing distributed log split in + logDirs);
...
{code}
...which kept accumulating on heap.
Now these monitored tasks get cleaned only when someone looks at the service
UI otherwise they keep growing boundless.
The postmortem of the heap dump showed that these task object occupied 99% of
HBase master heap accumulated over a week.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10051) rolling-restart.sh have old value of zookeeper.znode.unassiged config causing infinite loop
[
https://issues.apache.org/jira/browse/HBASE-10051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834747#comment-13834747
]
Hudson commented on HBASE-10051:
FAILURE: Integrated in hbase-0.96-hadoop2 #134 (See
[https://builds.apache.org/job/hbase-0.96-hadoop2/134/])
HBASE-10051 rolling-restart.sh have old value of zookeeper.znode.unassiged
config causing infinite loop (stack: rev 1546214)
* /hbase/branches/0.96/bin/rolling-restart.sh
rolling-restart.sh have old value of zookeeper.znode.unassiged config
causing infinite loop
-
Key: HBASE-10051
URL: https://issues.apache.org/jira/browse/HBASE-10051
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.96.0
Environment: x86_64 GNU/Linux
Reporter: Samir Ahmic
Assignee: Samir Ahmic
Fix For: 0.98.0, 0.96.1
Attachments: HBASE-10051.patch
While running rolling-restart.sh without arguments or with --master-only
argument script throws error when trying to check did all regions are cleared
from transition state. Here is line causing error:
{code}
if [ $zunassigned == null ]; then zunassigned=unassigned; fi
{code}
AFAIK default value of zookeeper.znode.unassiged is changed to
region-in-transition. So we should change this line to:
{code}
if [ $zunassigned == null ]; then zunassigned=region-in-transition; fi
{code}
I will attach patch today.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10050) graceful_stop.sh syntax error
[
https://issues.apache.org/jira/browse/HBASE-10050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834746#comment-13834746
]
Hudson commented on HBASE-10050:
FAILURE: Integrated in hbase-0.96-hadoop2 #134 (See
[https://builds.apache.org/job/hbase-0.96-hadoop2/134/])
HBASE-10050 graceful_stop.sh syntax error (stack: rev 1546213)
* /hbase/branches/0.96/bin/graceful_stop.sh
graceful_stop.sh syntax error
-
Key: HBASE-10050
URL: https://issues.apache.org/jira/browse/HBASE-10050
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.96.0
Environment: x86_64 GNU/Linux
Reporter: Samir Ahmic
Fix For: 0.98.0, 0.96.1
Attachments: HBASE-10050.patch
When i was trying to run graceful_stop.sh i got error:
{code}
./graceful_stop.sh: line 59: syntax error near unexpected token `;'
./graceful_stop.sh: line 59: `--failfast) ;'
{code}
After looking at script i notice that this lines are causing error:
{code}
--failfast) ;
-e) failfast=true; shift;;
--debug) ;
-d) debug=--debug; shift;;
{code}
They sholud be changed to this:
{code}
--failfast | -e) failfast=true; shift;;
--debug | -d) debug=--debug; shift;;
{code}
I will attach patch today.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10020) Add maven compile-protobuf profile
[
https://issues.apache.org/jira/browse/HBASE-10020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834745#comment-13834745
]
Hudson commented on HBASE-10020:
FAILURE: Integrated in hbase-0.96-hadoop2 #134 (See
[https://builds.apache.org/job/hbase-0.96-hadoop2/134/])
HBASE-10020 Add maven compile-protobuf profile (enis: rev 1546241)
* /hbase/branches/0.96/dev-support/build-proto.sh
* /hbase/branches/0.96/hbase-examples/pom.xml
*
/hbase/branches/0.96/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/generated/BulkDeleteProtos.java
*
/hbase/branches/0.96/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/generated/ExampleProtos.java
* /hbase/branches/0.96/hbase-protocol/README.txt
* /hbase/branches/0.96/hbase-protocol/pom.xml
* /hbase/branches/0.96/hbase-server/pom.xml
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/ColumnSchemaMessage.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/ScannerMessage.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/StorageClusterStatusMessage.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/TableInfoMessage.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/TableSchemaMessage.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/VersionMessage.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/protobuf/generated/PingProtos.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/protobuf/generated/TestProtos.java
* /hbase/branches/0.96/pom.xml
* /hbase/branches/0.96/src/main/docbkx/developer.xml
Add maven compile-protobuf profile
--
Key: HBASE-10020
URL: https://issues.apache.org/jira/browse/HBASE-10020
Project: HBase
Issue Type: Improvement
Components: build
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.0, 0.96.1
Attachments: hbase-10020_v1.patch, hbase-10020_v2.patch
Ad a maven profile to compile protobufs instead of the dev-support script
which is only for hbase-protocol module.
{code}
mvn test-compile -Dcompile-protobuf
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10053) ProtobufUtil#toMutationNoData interface change causes downstream project build break
[
https://issues.apache.org/jira/browse/HBASE-10053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834748#comment-13834748
]
Hudson commented on HBASE-10053:
FAILURE: Integrated in hbase-0.96-hadoop2 #134 (See
[https://builds.apache.org/job/hbase-0.96-hadoop2/134/])
hbase-10053: ProtobufUtil#toMutationNoData interface change causes downstream
project build break (jeffreyz: rev 1546249)
*
/hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
ProtobufUtil#toMutationNoData interface change causes downstream project
build break
Key: HBASE-10053
URL: https://issues.apache.org/jira/browse/HBASE-10053
Project: HBase
Issue Type: Bug
Reporter: Jeffrey Zhong
Assignee: Jeffrey Zhong
Priority: Critical
Fix For: 0.98.0, 0.96.1
Attachments: hbase-10053.patch
{code}
public static MutationProto toMutationNoData(final MutationType type, final
Mutation mutation)
{code}
is gone which causes Hive build break.
{code}
[javac]
.../hbase-handler/src/java/org/apache/hadoop/hive/hbase/PutWritable.java:65:
toMutationNoData(org.apache.hadoop.hbase.protobuf.generated.ClientProtos.MutationProto.MutationType,org.apache.hadoop.hbase.client.Mutation,org.apache.hadoop.hbase.protobuf.generated.ClientProtos.MutationProto.Builder)
in org.apache.hadoop.hbase.protobuf.ProtobufUtil cannot be applied to
(org.apache.hadoop.hbase.protobuf.generated.ClientProtos.MutationProto.MutationType,org.apache.hadoop.hbase.client.Put)
[javac] ProtobufUtil.toMutationNoData(MutationType.PUT,
put).writeDelimitedTo(DataOutputOutputStream.from(out));
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9892) Add info port to ServerName to support multi instances in a node
[ https://issues.apache.org/jira/browse/HBASE-9892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834752#comment-13834752 ] Liu Shaohui commented on HBASE-9892: [~stack] [~enis] Please help to review the patch for 0.94 again? If there is no other problem, I will start to do the patch for trunk. Happy Thanksgiving! Add info port to ServerName to support multi instances in a node Key: HBASE-9892 URL: https://issues.apache.org/jira/browse/HBASE-9892 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-9892-0.94-v1.diff, HBASE-9892-0.94-v2.diff, HBASE-9892-0.94-v3.diff, HBASE-9892-0.94-v4.diff The full GC time of regionserver with big heap( 30G ) usually can not be controlled in 30s. At the same time, the servers with 64G memory are normal. So we try to deploy multi rs instances(2-3 ) in a single node and the heap of each rs is about 20G ~ 24G. Most of the things works fine, except the hbase web ui. The master get the RS info port from conf, which is suitable for this situation of multi rs instances in a node. So we add info port to ServerName. a. at the startup, rs report it's info port to Hmaster. b, For root region, rs write the servername with info port ro the zookeeper root-region-server node. c, For meta regions, rs write the servername with info port to root region d. For user regions, rs write the servername with info port to meta regions So hmaster and client can get info port from the servername. To test this feature, I change the rs num from 1 to 3 in standalone mode, so we can test it in standalone mode, I think Hoya(hbase on yarn) will encounter the same problem. Anyone knows how Hoya handle this problem? PS: There are different formats for servername in zk node and meta table, i think we need to unify it and refactor the code. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-10054) Add the default column compression option
Liu Shaohui created HBASE-10054: --- Summary: Add the default column compression option Key: HBASE-10054 URL: https://issues.apache.org/jira/browse/HBASE-10054 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Add the default column compression option for cluster level. If users don't set column compression for a column family, we should use the default column compression. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8912) [0.94] AssignmentManager throws IllegalStateException from PENDING_OPEN to OFFLINE
[
https://issues.apache.org/jira/browse/HBASE-8912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834765#comment-13834765
]
chenjiajun commented on HBASE-8912:
---
the error would seem to disappeared on my cluster by the follow steps :
1 stop the hbase cluster
2 remove/mv the data of zookeeper
3 start the cluster
otherwise there is a solution for somebody's problem
:http://stackoverflow.com/questions/17792619/fatal-master-hmaster-unexpected-state-cannot-transit-it-to-offline
[0.94] AssignmentManager throws IllegalStateException from PENDING_OPEN to
OFFLINE
--
Key: HBASE-8912
URL: https://issues.apache.org/jira/browse/HBASE-8912
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Fix For: 0.94.15
Attachments: HBase-0.94 #1036 test - testRetrying [Jenkins].html
AM throws this exception which subsequently causes the master to abort:
{code}
java.lang.IllegalStateException: Unexpected state :
testRetrying,jjj,1372891751115.9b828792311001062a5ff4b1038fe33b.
state=PENDING_OPEN, ts=1372891751912,
server=hemera.apache.org,39064,1372891746132 .. Cannot transit it to OFFLINE.
at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
{code}
This exception trace is from the failing test TestMetaReaderEditor which is
failing pretty frequently, but looking at the test code, I think this is not
a test-only issue, but affects the main code path.
https://builds.apache.org/job/HBase-0.94/1036/testReport/junit/org.apache.hadoop.hbase.catalog/TestMetaReaderEditor/testRetrying/
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-10054) Add the default column compression option
[ https://issues.apache.org/jira/browse/HBASE-10054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-10054: Attachment: HBASE-10054-0.94-v1.diff Patch for 0.94 Add the default column compression option - Key: HBASE-10054 URL: https://issues.apache.org/jira/browse/HBASE-10054 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-10054-0.94-v1.diff Add the default column compression option for cluster level. If users don't set column compression for a column family, we should use the default column compression. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9884) Add Thrift and REST support for Visibility Labels
[
https://issues.apache.org/jira/browse/HBASE-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834769#comment-13834769
]
Hadoop QA commented on HBASE-9884:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616228/HBASE-9884_2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestAdmin
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8023//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8023//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8023//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8023//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8023//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8023//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8023//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8023//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8023//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8023//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8023//console
This message is automatically generated.
Add Thrift and REST support for Visibility Labels
-
Key: HBASE-9884
URL: https://issues.apache.org/jira/browse/HBASE-9884
Project: HBase
Issue Type: Improvement
Components: Client
Affects Versions: 0.98.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.98.0
Attachments: HBASE-9884.patch, HBASE-9884_1.patch, HBASE-9884_2.patch
In HBASE-7663 the REST and thrift support has been seperated out because the
patch is becoming bigger. This JIRA is to add the Thrift and REST part as a
seperated patch.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HBASE-10055) Add option to limit the scan speed in CopyTable and VerifyReplication
Liu Shaohui created HBASE-10055: --- Summary: Add option to limit the scan speed in CopyTable and VerifyReplication Key: HBASE-10055 URL: https://issues.apache.org/jira/browse/HBASE-10055 Project: HBase Issue Type: Improvement Environment: Add option to limit the scan speed in CopyTable and VerifyReplication. When adding a new replication, we use 'CopyTable' to copy old data from online cluster to peer cluster. After that, we use 'VerifyReplication' to check the data consistency of two clusters. To reduce the impact the online cluster's service, we add an option to limit the scan speed. Reporter: Liu Shaohui Priority: Minor -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-10055) Add option to limit the scan speed in CopyTable and VerifyReplication
[ https://issues.apache.org/jira/browse/HBASE-10055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-10055: Environment: was: Add option to limit the scan speed in CopyTable and VerifyReplication. When adding a new replication, we use 'CopyTable' to copy old data from online cluster to peer cluster. After that, we use 'VerifyReplication' to check the data consistency of two clusters. To reduce the impact the online cluster's service, we add an option to limit the scan speed. Add option to limit the scan speed in CopyTable and VerifyReplication - Key: HBASE-10055 URL: https://issues.apache.org/jira/browse/HBASE-10055 Project: HBase Issue Type: Improvement Environment: Reporter: Liu Shaohui Priority: Minor -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-10055) Add option to limit the scan speed in CopyTable and VerifyReplication
[ https://issues.apache.org/jira/browse/HBASE-10055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-10055: Description: Add option to limit the scan speed in CopyTable and VerifyReplication. When adding a new replication, we use 'CopyTable' to copy old data from online cluster to peer cluster. After that, we use 'VerifyReplication' to check the data consistency of two clusters. To reduce the impact the online cluster's service, we add an option to limit the scan speed. Add option to limit the scan speed in CopyTable and VerifyReplication - Key: HBASE-10055 URL: https://issues.apache.org/jira/browse/HBASE-10055 Project: HBase Issue Type: Improvement Environment: Reporter: Liu Shaohui Priority: Minor Add option to limit the scan speed in CopyTable and VerifyReplication. When adding a new replication, we use 'CopyTable' to copy old data from online cluster to peer cluster. After that, we use 'VerifyReplication' to check the data consistency of two clusters. To reduce the impact the online cluster's service, we add an option to limit the scan speed. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10053) ProtobufUtil#toMutationNoData interface change causes downstream project build break
[
https://issues.apache.org/jira/browse/HBASE-10053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834788#comment-13834788
]
Hudson commented on HBASE-10053:
SUCCESS: Integrated in HBase-TRUNK #4701 (See
[https://builds.apache.org/job/HBase-TRUNK/4701/])
hbase-10053: ProtobufUtil#toMutationNoData interface change causes downstream
project build break (jeffreyz: rev 1546248)
*
/hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
ProtobufUtil#toMutationNoData interface change causes downstream project
build break
Key: HBASE-10053
URL: https://issues.apache.org/jira/browse/HBASE-10053
Project: HBase
Issue Type: Bug
Reporter: Jeffrey Zhong
Assignee: Jeffrey Zhong
Priority: Critical
Fix For: 0.98.0, 0.96.1
Attachments: hbase-10053.patch
{code}
public static MutationProto toMutationNoData(final MutationType type, final
Mutation mutation)
{code}
is gone which causes Hive build break.
{code}
[javac]
.../hbase-handler/src/java/org/apache/hadoop/hive/hbase/PutWritable.java:65:
toMutationNoData(org.apache.hadoop.hbase.protobuf.generated.ClientProtos.MutationProto.MutationType,org.apache.hadoop.hbase.client.Mutation,org.apache.hadoop.hbase.protobuf.generated.ClientProtos.MutationProto.Builder)
in org.apache.hadoop.hbase.protobuf.ProtobufUtil cannot be applied to
(org.apache.hadoop.hbase.protobuf.generated.ClientProtos.MutationProto.MutationType,org.apache.hadoop.hbase.client.Put)
[javac] ProtobufUtil.toMutationNoData(MutationType.PUT,
put).writeDelimitedTo(DataOutputOutputStream.from(out));
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10050) graceful_stop.sh syntax error
[
https://issues.apache.org/jira/browse/HBASE-10050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834786#comment-13834786
]
Hudson commented on HBASE-10050:
SUCCESS: Integrated in HBase-TRUNK #4701 (See
[https://builds.apache.org/job/HBase-TRUNK/4701/])
HBASE-10050 graceful_stop.sh syntax error (stack: rev 1546212)
* /hbase/trunk/bin/graceful_stop.sh
graceful_stop.sh syntax error
-
Key: HBASE-10050
URL: https://issues.apache.org/jira/browse/HBASE-10050
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.96.0
Environment: x86_64 GNU/Linux
Reporter: Samir Ahmic
Fix For: 0.98.0, 0.96.1
Attachments: HBASE-10050.patch
When i was trying to run graceful_stop.sh i got error:
{code}
./graceful_stop.sh: line 59: syntax error near unexpected token `;'
./graceful_stop.sh: line 59: `--failfast) ;'
{code}
After looking at script i notice that this lines are causing error:
{code}
--failfast) ;
-e) failfast=true; shift;;
--debug) ;
-d) debug=--debug; shift;;
{code}
They sholud be changed to this:
{code}
--failfast | -e) failfast=true; shift;;
--debug | -d) debug=--debug; shift;;
{code}
I will attach patch today.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10051) rolling-restart.sh have old value of zookeeper.znode.unassiged config causing infinite loop
[
https://issues.apache.org/jira/browse/HBASE-10051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834787#comment-13834787
]
Hudson commented on HBASE-10051:
SUCCESS: Integrated in HBase-TRUNK #4701 (See
[https://builds.apache.org/job/HBase-TRUNK/4701/])
HBASE-10051 rolling-restart.sh have old value of zookeeper.znode.unassiged
config causing infinite loop (stack: rev 1546215)
* /hbase/trunk/bin/rolling-restart.sh
rolling-restart.sh have old value of zookeeper.znode.unassiged config
causing infinite loop
-
Key: HBASE-10051
URL: https://issues.apache.org/jira/browse/HBASE-10051
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.96.0
Environment: x86_64 GNU/Linux
Reporter: Samir Ahmic
Assignee: Samir Ahmic
Fix For: 0.98.0, 0.96.1
Attachments: HBASE-10051.patch
While running rolling-restart.sh without arguments or with --master-only
argument script throws error when trying to check did all regions are cleared
from transition state. Here is line causing error:
{code}
if [ $zunassigned == null ]; then zunassigned=unassigned; fi
{code}
AFAIK default value of zookeeper.znode.unassiged is changed to
region-in-transition. So we should change this line to:
{code}
if [ $zunassigned == null ]; then zunassigned=region-in-transition; fi
{code}
I will attach patch today.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10020) Add maven compile-protobuf profile
[
https://issues.apache.org/jira/browse/HBASE-10020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834785#comment-13834785
]
Hudson commented on HBASE-10020:
SUCCESS: Integrated in HBase-TRUNK #4701 (See
[https://builds.apache.org/job/HBase-TRUNK/4701/])
HBASE-10020 Add maven compile-protobuf profile (enis: rev 1546237)
* /hbase/trunk/dev-support/build-proto.sh
* /hbase/trunk/hbase-examples/pom.xml
*
/hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/generated/BulkDeleteProtos.java
*
/hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/generated/ExampleProtos.java
* /hbase/trunk/hbase-protocol/README.txt
* /hbase/trunk/hbase-protocol/pom.xml
* /hbase/trunk/hbase-server/pom.xml
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/ColumnSchemaMessage.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/ScannerMessage.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/StorageClusterStatusMessage.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/TableInfoMessage.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/TableSchemaMessage.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/VersionMessage.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/protobuf/generated/PingProtos.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/protobuf/generated/TestProtos.java
* /hbase/trunk/pom.xml
* /hbase/trunk/src/main/docbkx/developer.xml
Add maven compile-protobuf profile
--
Key: HBASE-10020
URL: https://issues.apache.org/jira/browse/HBASE-10020
Project: HBase
Issue Type: Improvement
Components: build
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.0, 0.96.1
Attachments: hbase-10020_v1.patch, hbase-10020_v2.patch
Ad a maven profile to compile protobufs instead of the dev-support script
which is only for hbase-protocol module.
{code}
mvn test-compile -Dcompile-protobuf
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10054) Add the default column compression option
[ https://issues.apache.org/jira/browse/HBASE-10054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834805#comment-13834805 ] Jean-Marc Spaggiari commented on HBASE-10054: - I don't see any real issue with that. But you should also update the documentation too. Can you do a trunk patch to trigger Hadoop QA? Add the default column compression option - Key: HBASE-10054 URL: https://issues.apache.org/jira/browse/HBASE-10054 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-10054-0.94-v1.diff Add the default column compression option for cluster level. If users don't set column compression for a column family, we should use the default column compression. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-10055) Add option to limit the scan speed in CopyTable and VerifyReplication
[ https://issues.apache.org/jira/browse/HBASE-10055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-10055: Attachment: HBASE-10055-0.94-v1.diff Patch for 0.94 Add option to limit the scan speed in CopyTable and VerifyReplication - Key: HBASE-10055 URL: https://issues.apache.org/jira/browse/HBASE-10055 Project: HBase Issue Type: Improvement Environment: Reporter: Liu Shaohui Priority: Minor Attachments: HBASE-10055-0.94-v1.diff Add option to limit the scan speed in CopyTable and VerifyReplication. When adding a new replication, we use 'CopyTable' to copy old data from online cluster to peer cluster. After that, we use 'VerifyReplication' to check the data consistency of two clusters. To reduce the impact the online cluster's service, we add an option to limit the scan speed. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-10056) region_status.rb not adopted to 0.96 api
Samir Ahmic created HBASE-10056:
---
Summary: region_status.rb not adopted to 0.96 api
Key: HBASE-10056
URL: https://issues.apache.org/jira/browse/HBASE-10056
Project: HBase
Issue Type: Bug
Components: scripts
Environment: x86_64 GNU/Linux
Reporter: Samir Ahmic
Assignee: Samir Ahmic
Priority: Minor
When I tried to run this script agains 0.96 cluster i got this error:
{code}
$ hbase org.jruby.Main region_status.rb
NameError: uninitialized constant IOException
const_missing at org/jruby/RubyModule.java:2647
(root) at region_status.rb:104
{code}
This line is causing issue:
{code}
table = HTable.new config, '.META.'.to_java_bytes
{code}
I belive this should be changed to:
{code}
table = HTable.new config, 'hbase:meta'.to_java_bytes
{code}
Second issue was caused by this line
{code}
server_count =
MetaScanner::allTableRegions(config,$tablename.to_java_bytes,false).size()
{code}
MetaScanner.allTablesRegion() is changed in 0.96 so i changed few lines in
order to fix this:
{code}
if not $tablename.nil?
# Creating TableName object
$TableName = HTableDescriptor.new($tablename.to_java_bytes).getTableName()
end
while true
if $tablename.nil?
server_count = admin.getClusterStatus().getRegionsCount()
else
#Creating HConnection object
connection = HConnectionManager::getConnection(config);
server_count = MetaScanner::allTableRegions(config, connection, $TableName
,false).size()
end
print Region Status: #{server_count} / #{meta_count}\n
{code}
After this changes script is working against 0.96. Since i'm far away from
expert in jruby :D can someone review this fix ? I will attach patch.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-10056) region_status.rb not adopted to 0.96 api
[
https://issues.apache.org/jira/browse/HBASE-10056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samir Ahmic updated HBASE-10056:
Attachment: HBASE-10056.patch
Here is patch.
region_status.rb not adopted to 0.96 api
Key: HBASE-10056
URL: https://issues.apache.org/jira/browse/HBASE-10056
Project: HBase
Issue Type: Bug
Components: scripts
Environment: x86_64 GNU/Linux
Reporter: Samir Ahmic
Assignee: Samir Ahmic
Priority: Minor
Attachments: HBASE-10056.patch
When I tried to run this script agains 0.96 cluster i got this error:
{code}
$ hbase org.jruby.Main region_status.rb
NameError: uninitialized constant IOException
const_missing at org/jruby/RubyModule.java:2647
(root) at region_status.rb:104
{code}
This line is causing issue:
{code}
table = HTable.new config, '.META.'.to_java_bytes
{code}
I belive this should be changed to:
{code}
table = HTable.new config, 'hbase:meta'.to_java_bytes
{code}
Second issue was caused by this line
{code}
server_count =
MetaScanner::allTableRegions(config,$tablename.to_java_bytes,false).size()
{code}
MetaScanner.allTablesRegion() is changed in 0.96 so i changed few lines in
order to fix this:
{code}
if not $tablename.nil?
# Creating TableName object
$TableName = HTableDescriptor.new($tablename.to_java_bytes).getTableName()
end
while true
if $tablename.nil?
server_count = admin.getClusterStatus().getRegionsCount()
else
#Creating HConnection object
connection = HConnectionManager::getConnection(config);
server_count = MetaScanner::allTableRegions(config, connection,
$TableName ,false).size()
end
print Region Status: #{server_count} / #{meta_count}\n
{code}
After this changes script is working against 0.96. Since i'm far away from
expert in jruby :D can someone review this fix ? I will attach patch.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10055) Add option to limit the scan speed in CopyTable and VerifyReplication
[
https://issues.apache.org/jira/browse/HBASE-10055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834945#comment-13834945
]
ramkrishna.s.vasudevan commented on HBASE-10055:
Possible to extract
{code}
+ rowdone ++;
+ if (scanLimit 0) {
+long sleepTimeInMs = 1000 * rowdone / scanLimit -
(System.currentTimeMillis() - st);
+if (sleepTimeInMs 0) {
+ try {
+Thread.sleep(sleepTimeInMs);
+ } catch (InterruptedException e) {
+// do nothing.
+ }
+}
+ }
{code}
This to a common method? Else looks good to me.
Add option to limit the scan speed in CopyTable and VerifyReplication
-
Key: HBASE-10055
URL: https://issues.apache.org/jira/browse/HBASE-10055
Project: HBase
Issue Type: Improvement
Environment:
Reporter: Liu Shaohui
Priority: Minor
Attachments: HBASE-10055-0.94-v1.diff
Add option to limit the scan speed in CopyTable and VerifyReplication.
When adding a new replication, we use 'CopyTable' to copy old data from
online cluster to peer cluster. After that, we use 'VerifyReplication' to
check the data consistency of two clusters.
To reduce the impact the online cluster's service, we add an option to limit
the scan speed.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10054) Add the default column compression option
[ https://issues.apache.org/jira/browse/HBASE-10054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834950#comment-13834950 ] ramkrishna.s.vasudevan commented on HBASE-10054: This code won't do anything as there is no real default value for the compression. Add the default column compression option - Key: HBASE-10054 URL: https://issues.apache.org/jira/browse/HBASE-10054 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-10054-0.94-v1.diff Add the default column compression option for cluster level. If users don't set column compression for a column family, we should use the default column compression. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10049) Small improvments in region_mover.rb
[ https://issues.apache.org/jira/browse/HBASE-10049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834954#comment-13834954 ] ramkrishna.s.vasudevan commented on HBASE-10049: Patch looks good to me. +1. Small improvments in region_mover.rb Key: HBASE-10049 URL: https://issues.apache.org/jira/browse/HBASE-10049 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-10049-0.94-v1.diff, HBASE-10049-0.94-v2.diff, HBASE-10049-trunk-v1.diff We use region_mover.rb in the graceful upgrade of hbase cluster. Here are small improvements. a. remove the table.close(), because the htable could be reused. b. Add more info in the log of moving region. c. Add 20s sleep in load command to make sure the rs finished initialization of rpc server. There is a time gap between rs startup report and rpc server initialization. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-10049) Small improvments in region_mover.rb
[ https://issues.apache.org/jira/browse/HBASE-10049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-10049: --- Fix Version/s: 0.98.0 Small improvments in region_mover.rb Key: HBASE-10049 URL: https://issues.apache.org/jira/browse/HBASE-10049 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 0.98.0 Attachments: HBASE-10049-0.94-v1.diff, HBASE-10049-0.94-v2.diff, HBASE-10049-trunk-v1.diff We use region_mover.rb in the graceful upgrade of hbase cluster. Here are small improvements. a. remove the table.close(), because the htable could be reused. b. Add more info in the log of moving region. c. Add 20s sleep in load command to make sure the rs finished initialization of rpc server. There is a time gap between rs startup report and rpc server initialization. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10049) Small improvments in region_mover.rb
[ https://issues.apache.org/jira/browse/HBASE-10049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834961#comment-13834961 ] Ted Yu commented on HBASE-10049: Integrated to trunk. Thanks for the patch, Shaohui. Thanks for the review, Ram. Small improvments in region_mover.rb Key: HBASE-10049 URL: https://issues.apache.org/jira/browse/HBASE-10049 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 0.98.0 Attachments: HBASE-10049-0.94-v1.diff, HBASE-10049-0.94-v2.diff, HBASE-10049-trunk-v1.diff We use region_mover.rb in the graceful upgrade of hbase cluster. Here are small improvements. a. remove the table.close(), because the htable could be reused. b. Add more info in the log of moving region. c. Add 20s sleep in load command to make sure the rs finished initialization of rpc server. There is a time gap between rs startup report and rpc server initialization. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9489) Add cp hooks in online merge before and after setting PONR
[ https://issues.apache.org/jira/browse/HBASE-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rajeshbabu updated HBASE-9489: -- Attachment: HBASE-9489_v3.patch [~ghelmling] bq. Maybe something like preMergeCommit / postMergeCommit would be better? preMergeCommit / postMergeCommit names are very meaningful. used the same in the current patch. [~yuzhih...@gmail.com] bq. rsCoprocessorHost can be created after the call to createMergedRegion(). In createMergedRegion() method we are using rsCoprocessorHost, so initializing before it. Patch addressing the review comments. Thanks. Add cp hooks in online merge before and after setting PONR -- Key: HBASE-9489 URL: https://issues.apache.org/jira/browse/HBASE-9489 Project: HBase Issue Type: Sub-task Reporter: rajeshbabu Assignee: rajeshbabu Attachments: HBASE-9489.patch, HBASE-9489_v2.patch, HBASE-9489_v3.patch As we need to merge index region along with user region we need the hooks before and after setting PONR in region merge transtion. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10005) TestVisibilityLabels fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-10005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834968#comment-13834968
]
Anoop Sam John commented on HBASE-10005:
Committed the patch to Trunk. Keeping open for some more builds
TestVisibilityLabels fails occasionally
---
Key: HBASE-10005
URL: https://issues.apache.org/jira/browse/HBASE-10005
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Ted Yu
Assignee: Anoop Sam John
Priority: Blocker
Fix For: 0.98.0
Attachments: HBASE-10005.patch,
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels-output.txt
I got the following test failures running test suite on hadoop-2 where
distributed log replay was turned on :
{code}
testAddVisibilityLabelsOnRSRestart(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.019 sec FAILURE!
java.lang.AssertionError: The count should be 8 expected:8 but was:6
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testAddVisibilityLabelsOnRSRestart(TestVisibilityLabels.java:408)
...
testClearUserAuths(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.002 sec FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testClearUserAuths(TestVisibilityLabels.java:505)
{code}
Logs to be attached
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10005) TestVisibilityLabels fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-10005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834981#comment-13834981
]
Ted Yu commented on HBASE-10005:
There seems to be some misspelled method names:
{code}
ListString labels = getUserAuthsFromLablesTable(user);
private ListString getUserAuthsFromLablesTable(byte[] user) throws
IOException {
ListString currentAuths = this.getUserAuthsFromLablesTable(user);
hbase-server//src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java
public void testLablesWithCheckAndPut() throws Throwable {
public void testLablesWithIncrement() throws Throwable {
public void testLablesWithAppend() throws Throwable {
hbase-server//src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabels.java
{code}
Lable - Label
TestVisibilityLabels fails occasionally
---
Key: HBASE-10005
URL: https://issues.apache.org/jira/browse/HBASE-10005
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Ted Yu
Assignee: Anoop Sam John
Priority: Blocker
Fix For: 0.98.0
Attachments: HBASE-10005.patch,
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels-output.txt
I got the following test failures running test suite on hadoop-2 where
distributed log replay was turned on :
{code}
testAddVisibilityLabelsOnRSRestart(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.019 sec FAILURE!
java.lang.AssertionError: The count should be 8 expected:8 but was:6
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testAddVisibilityLabelsOnRSRestart(TestVisibilityLabels.java:408)
...
testClearUserAuths(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.002 sec FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testClearUserAuths(TestVisibilityLabels.java:505)
{code}
Logs to be attached
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10054) Add the default column compression option
[ https://issues.apache.org/jira/browse/HBASE-10054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835006#comment-13835006 ] Jean-Marc Spaggiari commented on HBASE-10054: - I guess that's the goal. If you don't configure any default value for the compression, then you keep the existing behavior. If you want to have a default value, then you add it in your configuration file. Add the default column compression option - Key: HBASE-10054 URL: https://issues.apache.org/jira/browse/HBASE-10054 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-10054-0.94-v1.diff Add the default column compression option for cluster level. If users don't set column compression for a column family, we should use the default column compression. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10049) Small improvments in region_mover.rb
[ https://issues.apache.org/jira/browse/HBASE-10049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835010#comment-13835010 ] stack commented on HBASE-10049: --- Please add to 0.96 and 0.94 branches also. This is a critical ops script. Small improvments in region_mover.rb Key: HBASE-10049 URL: https://issues.apache.org/jira/browse/HBASE-10049 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 0.98.0 Attachments: HBASE-10049-0.94-v1.diff, HBASE-10049-0.94-v2.diff, HBASE-10049-trunk-v1.diff We use region_mover.rb in the graceful upgrade of hbase cluster. Here are small improvements. a. remove the table.close(), because the htable could be reused. b. Add more info in the log of moving region. c. Add 20s sleep in load command to make sure the rs finished initialization of rpc server. There is a time gap between rs startup report and rpc server initialization. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-10005) TestVisibilityLabels fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-10005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-10005:
---
Attachment: HBASE-10005_addendum.patch
Committed this addendum to correct the spell. Thanks a lot for finding it out
Ted!
TestVisibilityLabels fails occasionally
---
Key: HBASE-10005
URL: https://issues.apache.org/jira/browse/HBASE-10005
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Ted Yu
Assignee: Anoop Sam John
Priority: Blocker
Fix For: 0.98.0
Attachments: HBASE-10005.patch, HBASE-10005_addendum.patch,
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels-output.txt
I got the following test failures running test suite on hadoop-2 where
distributed log replay was turned on :
{code}
testAddVisibilityLabelsOnRSRestart(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.019 sec FAILURE!
java.lang.AssertionError: The count should be 8 expected:8 but was:6
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testAddVisibilityLabelsOnRSRestart(TestVisibilityLabels.java:408)
...
testClearUserAuths(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.002 sec FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testClearUserAuths(TestVisibilityLabels.java:505)
{code}
Logs to be attached
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9535: --- Attachment: 9535.v3.patch Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack Attachments: 9535.v1.patch, 9535.v2.patch, 9535.v3.patch ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon reassigned HBASE-9535: -- Assignee: Nicolas Liochon (was: stack) Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: Nicolas Liochon Attachments: 9535.v1.patch, 9535.v2.patch, 9535.v3.patch ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9535: --- Status: Patch Available (was: Open) Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack Attachments: 9535.v1.patch, 9535.v2.patch, 9535.v3.patch ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10050) graceful_stop.sh syntax error
[
https://issues.apache.org/jira/browse/HBASE-10050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835013#comment-13835013
]
stack commented on HBASE-10050:
---
It looks like 0.94 doesn't need this script. It looks like this:
{code}
1 --thrift) thrift=true; shift;;
2 --rest) rest=true; shift;;
3 --restart) restart=true; shift;;
4 --reload) reload=true; shift;;
5 --debug)debug=--debug; shift;;
{code}
graceful_stop.sh syntax error
-
Key: HBASE-10050
URL: https://issues.apache.org/jira/browse/HBASE-10050
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.96.0
Environment: x86_64 GNU/Linux
Reporter: Samir Ahmic
Fix For: 0.98.0, 0.96.1
Attachments: HBASE-10050.patch
When i was trying to run graceful_stop.sh i got error:
{code}
./graceful_stop.sh: line 59: syntax error near unexpected token `;'
./graceful_stop.sh: line 59: `--failfast) ;'
{code}
After looking at script i notice that this lines are causing error:
{code}
--failfast) ;
-e) failfast=true; shift;;
--debug) ;
-d) debug=--debug; shift;;
{code}
They sholud be changed to this:
{code}
--failfast | -e) failfast=true; shift;;
--debug | -d) debug=--debug; shift;;
{code}
I will attach patch today.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-10049) Small improvments in region_mover.rb
[ https://issues.apache.org/jira/browse/HBASE-10049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-10049: --- Fix Version/s: 0.96.2 0.94.15 Small improvments in region_mover.rb Key: HBASE-10049 URL: https://issues.apache.org/jira/browse/HBASE-10049 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 0.98.0, 0.94.15, 0.96.2 Attachments: HBASE-10049-0.94-v1.diff, HBASE-10049-0.94-v2.diff, HBASE-10049-trunk-v1.diff We use region_mover.rb in the graceful upgrade of hbase cluster. Here are small improvements. a. remove the table.close(), because the htable could be reused. b. Add more info in the log of moving region. c. Add 20s sleep in load command to make sure the rs finished initialization of rpc server. There is a time gap between rs startup report and rpc server initialization. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HBASE-10056) region_status.rb not adopted to 0.96 api
[
https://issues.apache.org/jira/browse/HBASE-10056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-10056.
---
Resolution: Fixed
Fix Version/s: 0.96.2
0.98.0
Hadoop Flags: Reviewed
Thank you Samir. This is pretty bad. Maybe enough to sink the RC I put up
last night. Committed to trunk and to 0.96.
region_status.rb not adopted to 0.96 api
Key: HBASE-10056
URL: https://issues.apache.org/jira/browse/HBASE-10056
Project: HBase
Issue Type: Bug
Components: scripts
Environment: x86_64 GNU/Linux
Reporter: Samir Ahmic
Assignee: Samir Ahmic
Priority: Minor
Fix For: 0.98.0, 0.96.2
Attachments: HBASE-10056.patch
When I tried to run this script agains 0.96 cluster i got this error:
{code}
$ hbase org.jruby.Main region_status.rb
NameError: uninitialized constant IOException
const_missing at org/jruby/RubyModule.java:2647
(root) at region_status.rb:104
{code}
This line is causing issue:
{code}
table = HTable.new config, '.META.'.to_java_bytes
{code}
I belive this should be changed to:
{code}
table = HTable.new config, 'hbase:meta'.to_java_bytes
{code}
Second issue was caused by this line
{code}
server_count =
MetaScanner::allTableRegions(config,$tablename.to_java_bytes,false).size()
{code}
MetaScanner.allTablesRegion() is changed in 0.96 so i changed few lines in
order to fix this:
{code}
if not $tablename.nil?
# Creating TableName object
$TableName = HTableDescriptor.new($tablename.to_java_bytes).getTableName()
end
while true
if $tablename.nil?
server_count = admin.getClusterStatus().getRegionsCount()
else
#Creating HConnection object
connection = HConnectionManager::getConnection(config);
server_count = MetaScanner::allTableRegions(config, connection,
$TableName ,false).size()
end
print Region Status: #{server_count} / #{meta_count}\n
{code}
After this changes script is working against 0.96. Since i'm far away from
expert in jruby :D can someone review this fix ? I will attach patch.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10005) TestVisibilityLabels fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-10005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835018#comment-13835018
]
Hadoop QA commented on HBASE-10005:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616294/HBASE-10005_addendum.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8025//console
This message is automatically generated.
TestVisibilityLabels fails occasionally
---
Key: HBASE-10005
URL: https://issues.apache.org/jira/browse/HBASE-10005
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Ted Yu
Assignee: Anoop Sam John
Priority: Blocker
Fix For: 0.98.0
Attachments: HBASE-10005.patch, HBASE-10005_addendum.patch,
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels-output.txt
I got the following test failures running test suite on hadoop-2 where
distributed log replay was turned on :
{code}
testAddVisibilityLabelsOnRSRestart(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.019 sec FAILURE!
java.lang.AssertionError: The count should be 8 expected:8 but was:6
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testAddVisibilityLabelsOnRSRestart(TestVisibilityLabels.java:408)
...
testClearUserAuths(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.002 sec FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testClearUserAuths(TestVisibilityLabels.java:505)
{code}
Logs to be attached
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HBASE-10057) TestRestoreFlushSnapshotFromClient and TestRestoreSnapshotFromClient fail to finish occasionally
Lars Hofhansl created HBASE-10057:
-
Summary: TestRestoreFlushSnapshotFromClient and
TestRestoreSnapshotFromClient fail to finish occasionally
Key: HBASE-10057
URL: https://issues.apache.org/jira/browse/HBASE-10057
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Looking at the test code I see
{code}
HTable table = new HTable(...)
try {
... test code ...
} finally {
table.close();
}
{code}
The try-finally, while well meaning seems to hide the actual failure.
I'll just remove that.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[
https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835020#comment-13835020
]
Hadoop QA commented on HBASE-9535:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616295/9535.v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8026//console
This message is automatically generated.
Try a pool of direct byte buffers handling incoming ipc requests
Key: HBASE-9535
URL: https://issues.apache.org/jira/browse/HBASE-9535
Project: HBase
Issue Type: Brainstorming
Reporter: stack
Assignee: Nicolas Liochon
Attachments: 9535.v1.patch, 9535.v2.patch, 9535.v3.patch
ipc takes in a query by allocating a ByteBuffer of the size of the request
and then reading off the socket into this on-heap BB.
Experiment with keeping a pool of BBs so we have some buffer reuse to cut on
garbage generated. Could checkout from pool in RpcServer#Reader. Could
check back into the pool when Handler is done just before it queues the
response on the Responder's queue. We should be good since, at least for
now, kvs get copied up into MSLAB (not references) when data gets stuffed
into MemStore; this should make it so no references left over when we check
the BB back into the pool for use next time around.
If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes
time so if already allocated, should be good. GC of DBBs is a pain but if in
a pool, we shouldn't be wanting this to happen). The copy from socket to the
DBB will be off-heap (should be fast).
Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by
size (we might want to bypass the pool if an object is size N).
DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when
short-circuit reading. It'd be a bummer if we had to allocate big objects
on-heap. Would still be an improvement.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-10057) TestRestoreFlushSnapshotFromClient and TestRestoreSnapshotFromClient fail to finish occasionally
[
https://issues.apache.org/jira/browse/HBASE-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-10057:
--
Attachment: 10057-trunk.txt
Trunk patch. Mostly indentation changes. Also removed some unused imports.
TestRestoreFlushSnapshotFromClient and TestRestoreSnapshotFromClient fail to
finish occasionally
Key: HBASE-10057
URL: https://issues.apache.org/jira/browse/HBASE-10057
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Attachments: 10057-trunk.txt
Looking at the test code I see
{code}
HTable table = new HTable(...)
try {
... test code ...
} finally {
table.close();
}
{code}
The try-finally, while well meaning seems to hide the actual failure.
I'll just remove that.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-10057) TestRestoreFlushSnapshotFromClient and TestRestoreSnapshotFromClient fail to finish occasionally
[
https://issues.apache.org/jira/browse/HBASE-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-10057:
--
Attachment: 10057-0.94.txt
Same for 0.94.
This is an simple test-only change. Unless I hear objections I will just commit
this.
TestRestoreFlushSnapshotFromClient and TestRestoreSnapshotFromClient fail to
finish occasionally
Key: HBASE-10057
URL: https://issues.apache.org/jira/browse/HBASE-10057
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Attachments: 10057-0.94.txt, 10057-trunk.txt
Looking at the test code I see
{code}
HTable table = new HTable(...)
try {
... test code ...
} finally {
table.close();
}
{code}
The try-finally, while well meaning seems to hide the actual failure.
I'll just remove that.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9489) Add cp hooks in online merge before and after setting PONR
[
https://issues.apache.org/jira/browse/HBASE-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835025#comment-13835025
]
Hadoop QA commented on HBASE-9489:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616285/HBASE-9489_v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestSplitLogWorker
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8024//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8024//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8024//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8024//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8024//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8024//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8024//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8024//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8024//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8024//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8024//console
This message is automatically generated.
Add cp hooks in online merge before and after setting PONR
--
Key: HBASE-9489
URL: https://issues.apache.org/jira/browse/HBASE-9489
Project: HBase
Issue Type: Sub-task
Reporter: rajeshbabu
Assignee: rajeshbabu
Attachments: HBASE-9489.patch, HBASE-9489_v2.patch,
HBASE-9489_v3.patch
As we need to merge index region along with user region we need the hooks
before and after setting PONR in region merge transtion.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Resolved] (HBASE-10057) TestRestoreFlushSnapshotFromClient and TestRestoreSnapshotFromClient fail to finish occasionally
[
https://issues.apache.org/jira/browse/HBASE-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl resolved HBASE-10057.
---
Resolution: Fixed
Fix Version/s: 0.96.2
0.94.15
0.98.0
Assignee: Lars Hofhansl
TestRestoreFlushSnapshotFromClient and TestRestoreSnapshotFromClient fail to
finish occasionally
Key: HBASE-10057
URL: https://issues.apache.org/jira/browse/HBASE-10057
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Fix For: 0.98.0, 0.94.15, 0.96.2
Attachments: 10057-0.94.txt, 10057-trunk.txt
Looking at the test code I see
{code}
HTable table = new HTable(...)
try {
... test code ...
} finally {
table.close();
}
{code}
The try-finally, while well meaning seems to hide the actual failure.
I'll just remove that.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-4811) Support reverse Scan
[
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-4811:
--
Attachment: 4811-trunk-v29.patch
Rebased patch for QA run.
Support reverse Scan
Key: HBASE-4811
URL: https://issues.apache.org/jira/browse/HBASE-4811
Project: HBase
Issue Type: New Feature
Components: Client
Affects Versions: 0.20.6, 0.94.7
Reporter: John Carrino
Assignee: chunhui shen
Fix For: 0.98.0, 0.96.1, 0.94.15
Attachments: 4811-0.94-v22.txt, 4811-0.94-v23.txt, 4811-0.94-v3.txt,
4811-trunk-v10.txt, 4811-trunk-v29.patch, 4811-trunk-v5.patch,
HBase-4811-0.94-v2.txt, HBase-4811-0.94.3modified.txt, hbase-4811-0.94
v21.patch, hbase-4811-0.94-v24.patch, hbase-4811-trunkv1.patch,
hbase-4811-trunkv11.patch, hbase-4811-trunkv12.patch,
hbase-4811-trunkv13.patch, hbase-4811-trunkv14.patch,
hbase-4811-trunkv15.patch, hbase-4811-trunkv16.patch,
hbase-4811-trunkv17.patch, hbase-4811-trunkv18.patch,
hbase-4811-trunkv19.patch, hbase-4811-trunkv20.patch,
hbase-4811-trunkv21.patch, hbase-4811-trunkv24.patch,
hbase-4811-trunkv24.patch, hbase-4811-trunkv25.patch,
hbase-4811-trunkv26.patch, hbase-4811-trunkv27.patch,
hbase-4811-trunkv28.patch, hbase-4811-trunkv4.patch,
hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch,
hbase-4811-trunkv9.patch
Reversed scan means scan the rows backward.
And StartRow bigger than StopRow in a reversed scan.
For example, for the following rows:
aaa/c1:q1/value1
aaa/c1:q2/value2
bbb/c1:q1/value1
bbb/c1:q2/value2
ccc/c1:q1/value1
ccc/c1:q2/value2
ddd/c1:q1/value1
ddd/c1:q2/value2
eee/c1:q1/value1
eee/c1:q2/value2
you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
Scan scan = new Scan();
scan.setStartRow('ddd');
scan.setStopRow('bbb');
scan.setReversed(true);
for(Result result:htable.getScanner(scan)){
System.out.println(result);
}
Aslo you could do the reversed scan with shell like this:
hbase scan 'table',{REVERSED = true,STARTROW='ddd', STOPROW='bbb'}
And the output is:
ddd/c1:q1/value1
ddd/c1:q2/value2
ccc/c1:q1/value1
ccc/c1:q2/value2
NOTE: when setting reversed as true for a client scan, you must set the start
row, else will throw exception. Through {@link
Scan#createBiggestByteArray(int)},you could get a big enough byte array as
the start row
All the documentation I find about HBase says that if you want forward and
reverse scans you should just build 2 tables and one be ascending and one
descending. Is there a fundamental reason that HBase only supports forward
Scan? It seems like a lot of extra space overhead and coding overhead (to
keep them in sync) to support 2 tables.
I am assuming this has been discussed before, but I can't find the
discussions anywhere about it or why it would be infeasible.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-4811) Support reverse Scan
[
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835083#comment-13835083
]
Hadoop QA commented on HBASE-4811:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616305/4811-trunk-v29.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8027//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8027//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8027//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8027//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8027//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8027//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8027//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8027//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8027//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8027//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8027//console
This message is automatically generated.
Support reverse Scan
Key: HBASE-4811
URL: https://issues.apache.org/jira/browse/HBASE-4811
Project: HBase
Issue Type: New Feature
Components: Client
Affects Versions: 0.20.6, 0.94.7
Reporter: John Carrino
Assignee: chunhui shen
Fix For: 0.98.0, 0.96.1, 0.94.15
Attachments: 4811-0.94-v22.txt, 4811-0.94-v23.txt, 4811-0.94-v3.txt,
4811-trunk-v10.txt, 4811-trunk-v29.patch, 4811-trunk-v5.patch,
HBase-4811-0.94-v2.txt, HBase-4811-0.94.3modified.txt, hbase-4811-0.94
v21.patch, hbase-4811-0.94-v24.patch, hbase-4811-trunkv1.patch,
hbase-4811-trunkv11.patch, hbase-4811-trunkv12.patch,
hbase-4811-trunkv13.patch, hbase-4811-trunkv14.patch,
hbase-4811-trunkv15.patch, hbase-4811-trunkv16.patch,
hbase-4811-trunkv17.patch, hbase-4811-trunkv18.patch,
hbase-4811-trunkv19.patch, hbase-4811-trunkv20.patch,
hbase-4811-trunkv21.patch, hbase-4811-trunkv24.patch,
hbase-4811-trunkv24.patch, hbase-4811-trunkv25.patch,
hbase-4811-trunkv26.patch, hbase-4811-trunkv27.patch,
hbase-4811-trunkv28.patch, hbase-4811-trunkv4.patch,
hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch,
hbase-4811-trunkv9.patch
Reversed scan means scan the rows backward.
And StartRow bigger than StopRow in a reversed scan.
For example, for the following rows:
aaa/c1:q1/value1
aaa/c1:q2/value2
bbb/c1:q1/value1
bbb/c1:q2/value2
ccc/c1:q1/value1
ccc/c1:q2/value2
ddd/c1:q1/value1
ddd/c1:q2/value2
eee/c1:q1/value1
eee/c1:q2/value2
you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
Scan scan = new Scan();
scan.setStartRow('ddd');
scan.setStopRow('bbb');
scan.setReversed(true);
for(Result result:htable.getScanner(scan)){
System.out.println(result);
}
Aslo you could do the reversed scan with shell like this:
hbase
[jira] [Commented] (HBASE-10049) Small improvments in region_mover.rb
[ https://issues.apache.org/jira/browse/HBASE-10049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835101#comment-13835101 ] Hudson commented on HBASE-10049: FAILURE: Integrated in HBase-0.94-security #348 (See [https://builds.apache.org/job/HBase-0.94-security/348/]) HBASE-10049 Small improvments in region_mover.rb (tedyu: rev 1546417) * /hbase/branches/0.94/bin/region_mover.rb Small improvments in region_mover.rb Key: HBASE-10049 URL: https://issues.apache.org/jira/browse/HBASE-10049 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 0.98.0, 0.94.15, 0.96.2 Attachments: HBASE-10049-0.94-v1.diff, HBASE-10049-0.94-v2.diff, HBASE-10049-trunk-v1.diff We use region_mover.rb in the graceful upgrade of hbase cluster. Here are small improvements. a. remove the table.close(), because the htable could be reused. b. Add more info in the log of moving region. c. Add 20s sleep in load command to make sure the rs finished initialization of rpc server. There is a time gap between rs startup report and rpc server initialization. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10057) TestRestoreFlushSnapshotFromClient and TestRestoreSnapshotFromClient fail to finish occasionally
[
https://issues.apache.org/jira/browse/HBASE-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835102#comment-13835102
]
Hudson commented on HBASE-10057:
FAILURE: Integrated in HBase-0.94-security #348 (See
[https://builds.apache.org/job/HBase-0.94-security/348/])
HBASE-10057 TestRestoreFlushSnapshotFromClient and
TestRestoreSnapshotFromClient fail to finish occasionally (larsh: rev 1546443)
*
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java
*
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreFlushSnapshotFromClient.java
TestRestoreFlushSnapshotFromClient and TestRestoreSnapshotFromClient fail to
finish occasionally
Key: HBASE-10057
URL: https://issues.apache.org/jira/browse/HBASE-10057
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Fix For: 0.98.0, 0.94.15, 0.96.2
Attachments: 10057-0.94.txt, 10057-trunk.txt
Looking at the test code I see
{code}
HTable table = new HTable(...)
try {
... test code ...
} finally {
table.close();
}
{code}
The try-finally, while well meaning seems to hide the actual failure.
I'll just remove that.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10057) TestRestoreFlushSnapshotFromClient and TestRestoreSnapshotFromClient fail to finish occasionally
[
https://issues.apache.org/jira/browse/HBASE-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835132#comment-13835132
]
Hudson commented on HBASE-10057:
SUCCESS: Integrated in HBase-TRUNK #4702 (See
[https://builds.apache.org/job/HBase-TRUNK/4702/])
HBASE-10057 TestRestoreFlushSnapshotFromClient and
TestRestoreSnapshotFromClient fail to finish occasionally (larsh: rev 1546442)
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreFlushSnapshotFromClient.java
TestRestoreFlushSnapshotFromClient and TestRestoreSnapshotFromClient fail to
finish occasionally
Key: HBASE-10057
URL: https://issues.apache.org/jira/browse/HBASE-10057
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Fix For: 0.98.0, 0.94.15, 0.96.2
Attachments: 10057-0.94.txt, 10057-trunk.txt
Looking at the test code I see
{code}
HTable table = new HTable(...)
try {
... test code ...
} finally {
table.close();
}
{code}
The try-finally, while well meaning seems to hide the actual failure.
I'll just remove that.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10049) Small improvments in region_mover.rb
[ https://issues.apache.org/jira/browse/HBASE-10049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835130#comment-13835130 ] Hudson commented on HBASE-10049: SUCCESS: Integrated in HBase-TRUNK #4702 (See [https://builds.apache.org/job/HBase-TRUNK/4702/]) HBASE-10049 Small improvments in region_mover.rb (tedyu: rev 1546393) * /hbase/trunk/bin/region_mover.rb Small improvments in region_mover.rb Key: HBASE-10049 URL: https://issues.apache.org/jira/browse/HBASE-10049 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 0.98.0, 0.94.15, 0.96.2 Attachments: HBASE-10049-0.94-v1.diff, HBASE-10049-0.94-v2.diff, HBASE-10049-trunk-v1.diff We use region_mover.rb in the graceful upgrade of hbase cluster. Here are small improvements. a. remove the table.close(), because the htable could be reused. b. Add more info in the log of moving region. c. Add 20s sleep in load command to make sure the rs finished initialization of rpc server. There is a time gap between rs startup report and rpc server initialization. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-10056) region_status.rb not adopted to 0.96 api
[
https://issues.apache.org/jira/browse/HBASE-10056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835131#comment-13835131
]
Hudson commented on HBASE-10056:
SUCCESS: Integrated in HBase-TRUNK #4702 (See
[https://builds.apache.org/job/HBase-TRUNK/4702/])
HBASE-10056 region_status.rb not adopted to 0.96 api (stack: rev 1546416)
* /hbase/trunk/bin/region_status.rb
region_status.rb not adopted to 0.96 api
Key: HBASE-10056
URL: https://issues.apache.org/jira/browse/HBASE-10056
Project: HBase
Issue Type: Bug
Components: scripts
Environment: x86_64 GNU/Linux
Reporter: Samir Ahmic
Assignee: Samir Ahmic
Priority: Minor
Fix For: 0.98.0, 0.96.2
Attachments: HBASE-10056.patch
When I tried to run this script agains 0.96 cluster i got this error:
{code}
$ hbase org.jruby.Main region_status.rb
NameError: uninitialized constant IOException
const_missing at org/jruby/RubyModule.java:2647
(root) at region_status.rb:104
{code}
This line is causing issue:
{code}
table = HTable.new config, '.META.'.to_java_bytes
{code}
I belive this should be changed to:
{code}
table = HTable.new config, 'hbase:meta'.to_java_bytes
{code}
Second issue was caused by this line
{code}
server_count =
MetaScanner::allTableRegions(config,$tablename.to_java_bytes,false).size()
{code}
MetaScanner.allTablesRegion() is changed in 0.96 so i changed few lines in
order to fix this:
{code}
if not $tablename.nil?
# Creating TableName object
$TableName = HTableDescriptor.new($tablename.to_java_bytes).getTableName()
end
while true
if $tablename.nil?
server_count = admin.getClusterStatus().getRegionsCount()
else
#Creating HConnection object
connection = HConnectionManager::getConnection(config);
server_count = MetaScanner::allTableRegions(config, connection,
$TableName ,false).size()
end
print Region Status: #{server_count} / #{meta_count}\n
{code}
After this changes script is working against 0.96. Since i'm far away from
expert in jruby :D can someone review this fix ? I will attach patch.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10005) TestVisibilityLabels fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-10005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835133#comment-13835133
]
Hudson commented on HBASE-10005:
SUCCESS: Integrated in HBase-TRUNK #4702 (See
[https://builds.apache.org/job/HBase-TRUNK/4702/])
HBASE-10005 TestVisibilityLabels fails occasionally (addendum) (anoopsamjohn:
rev 1546408)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabels.java
HBASE-10005 TestVisibilityLabels fails occasionally (anoopsamjohn: rev 1546396)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabels.java
TestVisibilityLabels fails occasionally
---
Key: HBASE-10005
URL: https://issues.apache.org/jira/browse/HBASE-10005
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Ted Yu
Assignee: Anoop Sam John
Priority: Blocker
Fix For: 0.98.0
Attachments: HBASE-10005.patch, HBASE-10005_addendum.patch,
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels-output.txt
I got the following test failures running test suite on hadoop-2 where
distributed log replay was turned on :
{code}
testAddVisibilityLabelsOnRSRestart(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.019 sec FAILURE!
java.lang.AssertionError: The count should be 8 expected:8 but was:6
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testAddVisibilityLabelsOnRSRestart(TestVisibilityLabels.java:408)
...
testClearUserAuths(org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels)
Time elapsed: 0.002 sec FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels.testClearUserAuths(TestVisibilityLabels.java:505)
{code}
Logs to be attached
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-4811) Support reverse Scan
[
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835135#comment-13835135
]
chunhui shen commented on HBASE-4811:
-
Will commit this to trunk if no objection
Support reverse Scan
Key: HBASE-4811
URL: https://issues.apache.org/jira/browse/HBASE-4811
Project: HBase
Issue Type: New Feature
Components: Client
Affects Versions: 0.20.6, 0.94.7
Reporter: John Carrino
Assignee: chunhui shen
Fix For: 0.98.0, 0.96.1, 0.94.15
Attachments: 4811-0.94-v22.txt, 4811-0.94-v23.txt, 4811-0.94-v3.txt,
4811-trunk-v10.txt, 4811-trunk-v29.patch, 4811-trunk-v5.patch,
HBase-4811-0.94-v2.txt, HBase-4811-0.94.3modified.txt, hbase-4811-0.94
v21.patch, hbase-4811-0.94-v24.patch, hbase-4811-trunkv1.patch,
hbase-4811-trunkv11.patch, hbase-4811-trunkv12.patch,
hbase-4811-trunkv13.patch, hbase-4811-trunkv14.patch,
hbase-4811-trunkv15.patch, hbase-4811-trunkv16.patch,
hbase-4811-trunkv17.patch, hbase-4811-trunkv18.patch,
hbase-4811-trunkv19.patch, hbase-4811-trunkv20.patch,
hbase-4811-trunkv21.patch, hbase-4811-trunkv24.patch,
hbase-4811-trunkv24.patch, hbase-4811-trunkv25.patch,
hbase-4811-trunkv26.patch, hbase-4811-trunkv27.patch,
hbase-4811-trunkv28.patch, hbase-4811-trunkv4.patch,
hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch,
hbase-4811-trunkv9.patch
Reversed scan means scan the rows backward.
And StartRow bigger than StopRow in a reversed scan.
For example, for the following rows:
aaa/c1:q1/value1
aaa/c1:q2/value2
bbb/c1:q1/value1
bbb/c1:q2/value2
ccc/c1:q1/value1
ccc/c1:q2/value2
ddd/c1:q1/value1
ddd/c1:q2/value2
eee/c1:q1/value1
eee/c1:q2/value2
you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
Scan scan = new Scan();
scan.setStartRow('ddd');
scan.setStopRow('bbb');
scan.setReversed(true);
for(Result result:htable.getScanner(scan)){
System.out.println(result);
}
Aslo you could do the reversed scan with shell like this:
hbase scan 'table',{REVERSED = true,STARTROW='ddd', STOPROW='bbb'}
And the output is:
ddd/c1:q1/value1
ddd/c1:q2/value2
ccc/c1:q1/value1
ccc/c1:q2/value2
NOTE: when setting reversed as true for a client scan, you must set the start
row, else will throw exception. Through {@link
Scan#createBiggestByteArray(int)},you could get a big enough byte array as
the start row
All the documentation I find about HBase says that if you want forward and
reverse scans you should just build 2 tables and one be ascending and one
descending. Is there a fundamental reason that HBase only supports forward
Scan? It seems like a lot of extra space overhead and coding overhead (to
keep them in sync) to support 2 tables.
I am assuming this has been discussed before, but I can't find the
discussions anywhere about it or why it would be infeasible.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10048) Add hlog number metric in regionserver
[
https://issues.apache.org/jira/browse/HBASE-10048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835141#comment-13835141
]
chunhui shen commented on HBASE-10048:
--
{code}
+ // +1 for the current log
+ numHLogFiles = regionServer.hlog.getNumLogFiles() + 1;
{code}
Need +2 for MetaServer.
But I think no addition is acceptable here.
Total size of all logs seems useful too since one hlog would be very big in
normal case
Other looks good to me
Add hlog number metric in regionserver
--
Key: HBASE-10048
URL: https://issues.apache.org/jira/browse/HBASE-10048
Project: HBase
Issue Type: Improvement
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
Attachments: HBASE-10048-0.94-v1.diff, HBASE-10048-0.94-v2.diff,
HBASE-10048-trunk-v1.diff
Add hlog number metric in regionserver.
We can use this metric to alert about memstore flush because of too many
hlogs.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10054) Add the default column compression option
[
https://issues.apache.org/jira/browse/HBASE-10054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835143#comment-13835143
]
chunhui shen commented on HBASE-10054:
--
IMPO, other similar configs could be added like this if we do so. It seems not
a good start
e.g. 'DataBlockEncoding', 'CACHE_DATA_ON_WRITE' and other common options in
HTableDescriptor or HColumnDescriptor.
We could do such a action on application code/script.
Thus, +0 to add this option for me.
About the patch,
Nit comments:
{code}
+ LOG.info(User donot set the compression for column fammily:
{code}
donot = don't
Using desc#setCompressionType instead of desc#setValue because the former will
skip the invalid compression type.
Before set the compression to hcolumndescriptor, check whether the compression
is available on the server, just like we do in HMaster#checkCompression
Add the default column compression option
-
Key: HBASE-10054
URL: https://issues.apache.org/jira/browse/HBASE-10054
Project: HBase
Issue Type: Improvement
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
Attachments: HBASE-10054-0.94-v1.diff
Add the default column compression option for cluster level.
If users don't set column compression for a column family, we should use the
default column compression.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10057) TestRestoreFlushSnapshotFromClient and TestRestoreSnapshotFromClient fail to finish occasionally
[
https://issues.apache.org/jira/browse/HBASE-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835148#comment-13835148
]
Hudson commented on HBASE-10057:
SUCCESS: Integrated in HBase-0.94 #1214 (See
[https://builds.apache.org/job/HBase-0.94/1214/])
HBASE-10057 TestRestoreFlushSnapshotFromClient and
TestRestoreSnapshotFromClient fail to finish occasionally (larsh: rev 1546443)
*
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java
*
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreFlushSnapshotFromClient.java
TestRestoreFlushSnapshotFromClient and TestRestoreSnapshotFromClient fail to
finish occasionally
Key: HBASE-10057
URL: https://issues.apache.org/jira/browse/HBASE-10057
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Fix For: 0.98.0, 0.94.15, 0.96.2
Attachments: 10057-0.94.txt, 10057-trunk.txt
Looking at the test code I see
{code}
HTable table = new HTable(...)
try {
... test code ...
} finally {
table.close();
}
{code}
The try-finally, while well meaning seems to hide the actual failure.
I'll just remove that.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-10049) Small improvments in region_mover.rb
[ https://issues.apache.org/jira/browse/HBASE-10049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835147#comment-13835147 ] Hudson commented on HBASE-10049: SUCCESS: Integrated in HBase-0.94 #1214 (See [https://builds.apache.org/job/HBase-0.94/1214/]) HBASE-10049 Small improvments in region_mover.rb (tedyu: rev 1546417) * /hbase/branches/0.94/bin/region_mover.rb Small improvments in region_mover.rb Key: HBASE-10049 URL: https://issues.apache.org/jira/browse/HBASE-10049 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 0.98.0, 0.94.15, 0.96.2 Attachments: HBASE-10049-0.94-v1.diff, HBASE-10049-0.94-v2.diff, HBASE-10049-trunk-v1.diff We use region_mover.rb in the graceful upgrade of hbase cluster. Here are small improvements. a. remove the table.close(), because the htable could be reused. b. Add more info in the log of moving region. c. Add 20s sleep in load command to make sure the rs finished initialization of rpc server. There is a time gap between rs startup report and rpc server initialization. -- This message was sent by Atlassian JIRA (v6.1#6144)
