[jira] [Commented] (HBASE-6188) Remove the concept of table owner
[ https://issues.apache.org/jira/browse/HBASE-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294183#comment-13294183 ] Hadoop QA commented on HBASE-6188: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12531920/HBASE-6188.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2160//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2160//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2160//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2160//console This message is automatically generated. Remove the concept of table owner - Key: HBASE-6188 URL: https://issues.apache.org/jira/browse/HBASE-6188 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.94.0, 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Laxman Labels: security Fix For: 0.96.0, 0.94.1 Attachments: HBASE-6188.1.patch, HBASE-6188.patch The table owner concept was a design simplification in the initial drop. First, the design changes under review means only a user with GLOBAL CREATE permission can create a table, which will probably be an administrator. Then, granting implicit permissions may lead to oversights and it adds unnecessary conditionals to our code. So instead the administrator with GLOBAL CREATE permission should make the appropriate grants at table create time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6185) Update javadoc for ConstantSizeRegionSplitPolicy class
[
https://issues.apache.org/jira/browse/HBASE-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294193#comment-13294193
]
Hadoop QA commented on HBASE-6185:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12531923/HBASE-6185.v3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+0 tests included. The patch appears to be a documentation patch that
doesn't require tests.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
-1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2161//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2161//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2161//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2161//console
This message is automatically generated.
Update javadoc for ConstantSizeRegionSplitPolicy class
--
Key: HBASE-6185
URL: https://issues.apache.org/jira/browse/HBASE-6185
Project: HBase
Issue Type: Bug
Components: documentation
Affects Versions: 0.94.0
Reporter: nneverwei
Fix For: 0.94.1
Attachments: HBASE-6185.patch, HBASE-6185.v2.patch,
HBASE-6185.v3.patch
When using hbase0.94.0 we met a strange problem.
We config the 'hbase.hregion.max.filesize' to 100Gb (The recommed value to
act as auto-split turn off).
{code:xml}
property
namehbase.hregion.max.filesize/name
value107374182400/value
/property
{code}
Then we keep putting datas into a table.
But when the data size far more less than 100Gb(about 500~600 uncompressed
datas), the table auto splte to 2 regions...
I change the log4j config to DEBUG, and saw logs below:
{code}
2012-06-07 10:30:52,161 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Finished memstore flush of ~128.0m/134221272, currentsize=1.5m/1617744 for
region FileStructIndex,,1339032525500.7b229abcd0785408251a579e9bdf49c8. in
3201ms, sequenceid=176387980, compaction requested=false
2012-06-07 10:30:52,161 DEBUG
org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy:
ShouldSplit because info size=138657416, sizeToCheck=134217728,
regionsWithCommonTable=1
2012-06-07 10:30:52,161 DEBUG
org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy:
ShouldSplit because info size=138657416, sizeToCheck=134217728,
regionsWithCommonTable=1
2012-06-07 10:30:52,240 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Split requested for
FileStructIndex,,1339032525500.7b229abcd0785408251a579e9bdf49c8..
compaction_queue=(0:0), split_queue=0
2012-06-07 10:30:52,265 INFO
org.apache.hadoop.hbase.regionserver.SplitTransaction: Starting split of
region FileStructIndex,,1339032525500.7b229abcd0785408251a579e9bdf49c8.
2012-06-07 10:30:52,265 DEBUG
org.apache.hadoop.hbase.regionserver.SplitTransaction:
regionserver:60020-0x137c4929efe0001 Creating ephemeral node for
7b229abcd0785408251a579e9bdf49c8 in SPLITTING state
2012-06-07 10:30:52,368 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x137c4929efe0001 Attempting to transition node
7b229abcd0785408251a579e9bdf49c8 from RS_ZK_REGION_SPLITTING to
RS_ZK_REGION_SPLITTING
2012-06-07 10:30:52,382 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x137c4929efe0001 Successfully transitioned node
7b229abcd0785408251a579e9bdf49c8 from RS_ZK_REGION_SPLITTING to
RS_ZK_REGION_SPLITTING
2012-06-07 10:30:52,410 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Closing FileStructIndex,,1339032525500.7b229abcd0785408251a579e9bdf49c8.:
disabling compactions flushes
2012-06-07 10:30:52,410 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer:
NotServingRegionException;
FileStructIndex,,1339032525500.7b229abcd0785408251a579e9bdf49c8. is closing
2012-06-07 10:30:52,411 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer:
NotServingRegionException;
FileStructIndex,,1339032525500.7b229abcd0785408251a579e9bdf49c8. is closing
{code}
{color:red}IncreasingToUpperBoundRegionSplitPolicy: ShouldSplit
[jira] [Commented] (HBASE-6134) Improvement for split-worker to speed up distributed log splitting
[ https://issues.apache.org/jira/browse/HBASE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294195#comment-13294195 ] Hudson commented on HBASE-6134: --- Integrated in HBase-TRUNK #3021 (See [https://builds.apache.org/job/HBase-TRUNK/3021/]) HBASE-6134 Improvement for split-worker to speed up distributed log splitting (Chunhui) (Revision 1349632) Result = SUCCESS tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java Improvement for split-worker to speed up distributed log splitting -- Key: HBASE-6134 URL: https://issues.apache.org/jira/browse/HBASE-6134 Project: HBase Issue Type: Improvement Components: wal Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.96.0 Attachments: 6134v4.patch, HBASE-6134.patch, HBASE-6134v2.patch, HBASE-6134v3-92.patch, HBASE-6134v3.patch, HBASE-6134v4.patch First，we do the test between local-master-splitting and distributed-log-splitting Environment：34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths splitting work), 400 regions in one hlog file local-master-split:60s+ distributed-log-splitting:165s+ In fact, in our production environment, distributed-log-splitting also took 60s with 30 regionservers for 34 hlog files (regionserver may be in high load) We found split-worker split one log file took about 20s (30ms~50ms per writer.close(); 10ms per create writers ) I think we could do the improvement for this: Parallelizing the create and close writers in threads In the patch, change the logic for distributed-log-splitting same as the local-master-splitting and parallelizing the close in threads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6188) Remove the concept of table owner
[ https://issues.apache.org/jira/browse/HBASE-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294228#comment-13294228 ] Laxman commented on HBASE-6188: --- Andy, Thanks for through review. Updated with my comments in review board. Remove the concept of table owner - Key: HBASE-6188 URL: https://issues.apache.org/jira/browse/HBASE-6188 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.94.0, 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Laxman Labels: security Fix For: 0.96.0, 0.94.1 Attachments: HBASE-6188.1.patch, HBASE-6188.patch The table owner concept was a design simplification in the initial drop. First, the design changes under review means only a user with GLOBAL CREATE permission can create a table, which will probably be an administrator. Then, granting implicit permissions may lead to oversights and it adds unnecessary conditionals to our code. So instead the administrator with GLOBAL CREATE permission should make the appropriate grants at table create time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog
[
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294282#comment-13294282
]
Xing Shi commented on HBASE-4487:
-
@[~yuzhih...@gmail.com] or @[~dhruba]
I have a question about the code in the trunk:
{code}
try {
Integer lid = getLock(lockid, row, true);
this.updatesLock.readLock().lock();
try {
} finally {
this.updatesLock.readLock().unlock();
releaseRowLock(lid);
}
if (writeToWAL) {
this.log.sync(txid); // sync the transaction log outside the rowlock
}
} finally {
closeRegionOperation();
}
{code}
What will happen if
{code}this.log.sync(txid);{code}
failed?
As I know, the RS will tell client that this op failed, and the client will
retry. So MemStore and hlog on HDFS are inconsistent.
Am I right? Or the RS exits if an HDFS write/sync fails as Dhruba said?
The increment operation can release the rowlock before sync-ing the Hlog
Key: HBASE-4487
URL: https://issues.apache.org/jira/browse/HBASE-4487
Project: HBase
Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
Fix For: 0.94.0
Attachments: 4487-v7.txt, appendNoSync4.txt, appendNoSync5.txt,
appendNoSync6.txt
This allows for better throughput when there are hot rows.I have seen this
change make a single row update improve from 400 increments/sec/server to
4000 increments/sec/server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6205) Support an option to keep data of dropped table for some time
chunhui shen created HBASE-6205: --- Summary: Support an option to keep data of dropped table for some time Key: HBASE-6205 URL: https://issues.apache.org/jira/browse/HBASE-6205 Project: HBase Issue Type: Improvement Components: master Reporter: chunhui shen Assignee: chunhui shen User may drop table accidentally because of error code or other uncertain reasons. Unfortunately, it happens in our environment because one user make a mistake between production cluster and testing cluster. So, I just give a suggest, do we need to support an option to keep data of dropped table for some time, eg.1 day. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6205) Support an option to keep data of dropped table for some time
[
https://issues.apache.org/jira/browse/HBASE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chunhui shen updated HBASE-6205:
Attachment: HBASE-6205.patch
In the patch:
Move regions to trash table dir when dropping tables, and clear these regions
after time out.
Default keep time: 1 day
{code}
this.keepTime = conf.getLong(hbase.master.trashtable.keeptime,
+24 * 3600 * 1000l);
{code}
Support an option to keep data of dropped table for some time
-
Key: HBASE-6205
URL: https://issues.apache.org/jira/browse/HBASE-6205
Project: HBase
Issue Type: Improvement
Components: master
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: HBASE-6205.patch
User may drop table accidentally because of error code or other uncertain
reasons.
Unfortunately, it happens in our environment because one user make a mistake
between production cluster and testing cluster.
So, I just give a suggest, do we need to support an option to keep data of
dropped table for some time, eg.1 day.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6205) Support an option to keep data of dropped table for some time
[ https://issues.apache.org/jira/browse/HBASE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-6205: Description: User may drop table accidentally because of error code or other uncertain reasons. Unfortunately, it happens in our environment because one user make a mistake between production cluster and testing cluster. So, I just give a suggestion, do we need to support an option to keep data of dropped table for some time, eg.1 day. was: User may drop table accidentally because of error code or other uncertain reasons. Unfortunately, it happens in our environment because one user make a mistake between production cluster and testing cluster. So, I just give a suggest, do we need to support an option to keep data of dropped table for some time, eg.1 day. Support an option to keep data of dropped table for some time - Key: HBASE-6205 URL: https://issues.apache.org/jira/browse/HBASE-6205 Project: HBase Issue Type: Improvement Components: master Reporter: chunhui shen Assignee: chunhui shen Attachments: HBASE-6205.patch User may drop table accidentally because of error code or other uncertain reasons. Unfortunately, it happens in our environment because one user make a mistake between production cluster and testing cluster. So, I just give a suggestion, do we need to support an option to keep data of dropped table for some time, eg.1 day. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6205) Support an option to keep data of dropped table for some time
[ https://issues.apache.org/jira/browse/HBASE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-6205: Issue Type: New Feature (was: Improvement) Support an option to keep data of dropped table for some time - Key: HBASE-6205 URL: https://issues.apache.org/jira/browse/HBASE-6205 Project: HBase Issue Type: New Feature Components: master Reporter: chunhui shen Assignee: chunhui shen Attachments: HBASE-6205.patch User may drop table accidentally because of error code or other uncertain reasons. Unfortunately, it happens in our environment because one user make a mistake between production cluster and testing cluster. So, I just give a suggestion, do we need to support an option to keep data of dropped table for some time, eg.1 day. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6197) HRegion's append operation will lost data
[ https://issues.apache.org/jira/browse/HBASE-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6197: -- Hadoop Flags: Reviewed Status: Patch Available (was: Open) HRegion's append operation will lost data - Key: HBASE-6197 URL: https://issues.apache.org/jira/browse/HBASE-6197 Project: HBase Issue Type: Bug Components: regionserver Reporter: Xing Shi Assignee: ShiXing Attachments: HBASE-6197-trunk-V1.patch Like the HBASE-6195, when flushing the append thread will read out the old value for the larger timestamp in snapshot and smaller timestamp in memstore. We Should make the first-in-thread generates the smaller timestamp. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6197) HRegion's append operation may lose data
[ https://issues.apache.org/jira/browse/HBASE-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6197: -- Summary: HRegion's append operation may lose data (was: HRegion's append operation will lost data) HRegion's append operation may lose data Key: HBASE-6197 URL: https://issues.apache.org/jira/browse/HBASE-6197 Project: HBase Issue Type: Bug Components: regionserver Reporter: Xing Shi Assignee: ShiXing Attachments: HBASE-6197-trunk-V1.patch Like the HBASE-6195, when flushing the append thread will read out the old value for the larger timestamp in snapshot and smaller timestamp in memstore. We Should make the first-in-thread generates the smaller timestamp. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6134) Improvement for split-worker to speed up distributed log splitting
[ https://issues.apache.org/jira/browse/HBASE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294394#comment-13294394 ] Hudson commented on HBASE-6134: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #52 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/52/]) HBASE-6134 Improvement for split-worker to speed up distributed log splitting (Chunhui) (Revision 1349632) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java Improvement for split-worker to speed up distributed log splitting -- Key: HBASE-6134 URL: https://issues.apache.org/jira/browse/HBASE-6134 Project: HBase Issue Type: Improvement Components: wal Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.96.0 Attachments: 6134v4.patch, HBASE-6134.patch, HBASE-6134v2.patch, HBASE-6134v3-92.patch, HBASE-6134v3.patch, HBASE-6134v4.patch First，we do the test between local-master-splitting and distributed-log-splitting Environment：34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths splitting work), 400 regions in one hlog file local-master-split:60s+ distributed-log-splitting:165s+ In fact, in our production environment, distributed-log-splitting also took 60s with 30 regionservers for 34 hlog files (regionserver may be in high load) We found split-worker split one log file took about 20s (30ms~50ms per writer.close(); 10ms per create writers ) I think we could do the improvement for this: Parallelizing the create and close writers in threads In the patch, change the logic for distributed-log-splitting same as the local-master-splitting and parallelizing the close in threads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6195) Increment data will be lost when the memstore is flushed
[ https://issues.apache.org/jira/browse/HBASE-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294395#comment-13294395 ] Hudson commented on HBASE-6195: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #52 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/52/]) HBASE-6195 Addednum changes the method variable to be consistent with test name (Xing Shi) (Revision 1349619) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java Increment data will be lost when the memstore is flushed Key: HBASE-6195 URL: https://issues.apache.org/jira/browse/HBASE-6195 Project: HBase Issue Type: Bug Components: regionserver Reporter: Xing Shi Assignee: ShiXing Fix For: 0.96.0, 0.94.1 Attachments: 6195-trunk-V7.patch, 6195.addendum, HBASE-6195-trunk-V2.patch, HBASE-6195-trunk-V3.patch, HBASE-6195-trunk-V4.patch, HBASE-6195-trunk-V5.patch, HBASE-6195-trunk-V6.patch, HBASE-6195-trunk.patch There are two problems in increment() now: First: I see that the timestamp(the variable now) in HRegion's Increment() is generated before got the rowLock, so when there are multi-thread increment the same row, although it generate earlier, it may got the lock later. Because increment just store one version, so till now, the result will still be right. When the region is flushing, these increment will read the kv from snapshot and memstore with whose timestamp is larger, and write it back to memstore. If the snapshot's timestamp larger than the memstore, the increment will got the old data and then do the increment, it's wrong. Secondly: Also there is a risk in increment. Because it writes the memstore first and then HLog, so if it writes HLog failed, the client will also read the incremented value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6092) Authorize flush, split, compact operations in AccessController
[ https://issues.apache.org/jira/browse/HBASE-6092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294396#comment-13294396 ] Hudson commented on HBASE-6092: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #52 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/52/]) HBASE-6092. Authorize flush, split, compact operations in AccessController (Laxman) (Revision 1349596) Result = FAILURE apurtell : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java Authorize flush, split, compact operations in AccessController -- Key: HBASE-6092 URL: https://issues.apache.org/jira/browse/HBASE-6092 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.94.0, 0.96.0, 0.94.1 Reporter: Laxman Assignee: Laxman Labels: acl, security Fix For: 0.96.0, 0.94.1 Attachments: HBASE-6092.1.patch, HBASE-6092.patch Currently, flush, split and compaction are not checked for authorization in AccessController. With the current implementation any unauthorized client can trigger these operations on a table. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6197) HRegion's append operation may lose data
[ https://issues.apache.org/jira/browse/HBASE-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294416#comment-13294416 ] Hadoop QA commented on HBASE-6197: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12531914/HBASE-6197-trunk-V1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.regionserver.wal.TestHLog Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2162//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2162//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2162//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2162//console This message is automatically generated. HRegion's append operation may lose data Key: HBASE-6197 URL: https://issues.apache.org/jira/browse/HBASE-6197 Project: HBase Issue Type: Bug Components: regionserver Reporter: Xing Shi Assignee: ShiXing Attachments: HBASE-6197-trunk-V1.patch Like the HBASE-6195, when flushing the append thread will read out the old value for the larger timestamp in snapshot and smaller timestamp in memstore. We Should make the first-in-thread generates the smaller timestamp. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6188) Remove the concept of table owner
[ https://issues.apache.org/jira/browse/HBASE-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294468#comment-13294468 ] Andrew Purtell commented on HBASE-6188: --- Posted a reply in review board. Remove the concept of table owner - Key: HBASE-6188 URL: https://issues.apache.org/jira/browse/HBASE-6188 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.94.0, 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Laxman Labels: security Fix For: 0.96.0, 0.94.1 Attachments: HBASE-6188.1.patch, HBASE-6188.patch The table owner concept was a design simplification in the initial drop. First, the design changes under review means only a user with GLOBAL CREATE permission can create a table, which will probably be an administrator. Then, granting implicit permissions may lead to oversights and it adds unnecessary conditionals to our code. So instead the administrator with GLOBAL CREATE permission should make the appropriate grants at table create time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5924) In the client code, don't wait for all the requests to be executed before resubmitting a request in error.
[ https://issues.apache.org/jira/browse/HBASE-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-5924: --- Status: Open (was: Patch Available) In the client code, don't wait for all the requests to be executed before resubmitting a request in error. -- Key: HBASE-5924 URL: https://issues.apache.org/jira/browse/HBASE-5924 Project: HBase Issue Type: Improvement Components: client Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Fix For: 0.96.0 Attachments: 5924.v11.patch, 5924.v14.patch, 5924.v5.patch, 5924.v9.patch The client (in the function HConnectionManager#processBatchCallback) works in two steps: - make the requests - collect the failures and successes and prepare for retry It means that when there is an immediate error (region moved, split, dead server, ...) we still wait for all the initial requests to be executed before submitting again the failed request. If we have a scenario with all the requests taking 5 seconds we have a final execution time of: 5 (initial requests) + 1 (wait time) + 5 (final request) = 11s. We could improve this by analyzing immediately the results. This would lead us, for the scenario mentioned above, to 6 seconds. So we could have a performance improvement of nearly 50% in many cases, and much more than 50% if the request execution time is different. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5924) In the client code, don't wait for all the requests to be executed before resubmitting a request in error.
[ https://issues.apache.org/jira/browse/HBASE-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-5924: --- Attachment: 5924.v19.patch In the client code, don't wait for all the requests to be executed before resubmitting a request in error. -- Key: HBASE-5924 URL: https://issues.apache.org/jira/browse/HBASE-5924 Project: HBase Issue Type: Improvement Components: client Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Fix For: 0.96.0 Attachments: 5924.v11.patch, 5924.v14.patch, 5924.v19.patch, 5924.v5.patch, 5924.v9.patch The client (in the function HConnectionManager#processBatchCallback) works in two steps: - make the requests - collect the failures and successes and prepare for retry It means that when there is an immediate error (region moved, split, dead server, ...) we still wait for all the initial requests to be executed before submitting again the failed request. If we have a scenario with all the requests taking 5 seconds we have a final execution time of: 5 (initial requests) + 1 (wait time) + 5 (final request) = 11s. We could improve this by analyzing immediately the results. This would lead us, for the scenario mentioned above, to 6 seconds. So we could have a performance improvement of nearly 50% in many cases, and much more than 50% if the request execution time is different. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6188) Remove the concept of table owner
[ https://issues.apache.org/jira/browse/HBASE-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294514#comment-13294514 ] Laxman commented on HBASE-6188: --- bq. 1) In PostCreate - We can grant CRWA permissions to the current user(i.e. owner). I was thinking to ignoring the owner and PostCreate will make use of getActiveUser() to get the requested user. But, as per your comments, i understand that, we still need to consider owner to make it backward compatible. bq. Forgot to mention that also this needs to happen if the table owner is changed via setOwner(). I think this needs to be handled in postModifyTable. But I can see that raises some more questions. * Should we revoke permissions for old owner? If yes, how do we track old owner in postModify? Please correct me if my understanding is incorrect. Remove the concept of table owner - Key: HBASE-6188 URL: https://issues.apache.org/jira/browse/HBASE-6188 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.94.0, 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Laxman Labels: security Fix For: 0.96.0, 0.94.1 Attachments: HBASE-6188.1.patch, HBASE-6188.patch The table owner concept was a design simplification in the initial drop. First, the design changes under review means only a user with GLOBAL CREATE permission can create a table, which will probably be an administrator. Then, granting implicit permissions may lead to oversights and it adds unnecessary conditionals to our code. So instead the administrator with GLOBAL CREATE permission should make the appropriate grants at table create time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6188) Remove the concept of table owner
[ https://issues.apache.org/jira/browse/HBASE-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laxman updated HBASE-6188: -- Status: Patch Available (was: Open) Remove the concept of table owner - Key: HBASE-6188 URL: https://issues.apache.org/jira/browse/HBASE-6188 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.94.0, 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Laxman Labels: security Fix For: 0.96.0, 0.94.1 Attachments: HBASE-6188.1.patch, HBASE-6188.2.patch, HBASE-6188.patch The table owner concept was a design simplification in the initial drop. First, the design changes under review means only a user with GLOBAL CREATE permission can create a table, which will probably be an administrator. Then, granting implicit permissions may lead to oversights and it adds unnecessary conditionals to our code. So instead the administrator with GLOBAL CREATE permission should make the appropriate grants at table create time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5924) In the client code, don't wait for all the requests to be executed before resubmitting a request in error.
[ https://issues.apache.org/jira/browse/HBASE-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294587#comment-13294587 ] nkeywal commented on HBASE-5924: v19 with all the comments taken into account. I will create an another jira to rearrange the coprocessors tests on the znode. In the client code, don't wait for all the requests to be executed before resubmitting a request in error. -- Key: HBASE-5924 URL: https://issues.apache.org/jira/browse/HBASE-5924 Project: HBase Issue Type: Improvement Components: client Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Fix For: 0.96.0 Attachments: 5924.v11.patch, 5924.v14.patch, 5924.v19.patch, 5924.v5.patch, 5924.v9.patch The client (in the function HConnectionManager#processBatchCallback) works in two steps: - make the requests - collect the failures and successes and prepare for retry It means that when there is an immediate error (region moved, split, dead server, ...) we still wait for all the initial requests to be executed before submitting again the failed request. If we have a scenario with all the requests taking 5 seconds we have a final execution time of: 5 (initial requests) + 1 (wait time) + 5 (final request) = 11s. We could improve this by analyzing immediately the results. This would lead us, for the scenario mentioned above, to 6 seconds. So we could have a performance improvement of nearly 50% in many cases, and much more than 50% if the request execution time is different. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6143) Make region assignment smarter when regions are re-enabled.
[ https://issues.apache.org/jira/browse/HBASE-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294618#comment-13294618 ] Nicolas Spiegelberg commented on HBASE-6143: The default behavior should most likely take load balancing into account (See HBASE-4191). In FB, our master has a concept of 'favored nodes' that are assigned upon Region creation and map directly to the HDFS replication pipeline. Make region assignment smarter when regions are re-enabled. --- Key: HBASE-6143 URL: https://issues.apache.org/jira/browse/HBASE-6143 Project: HBase Issue Type: Improvement Reporter: Elliott Clark Assignee: Elliott Clark Right now a random region server is picked when re-enabling a table. This could be much smarter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5924) In the client code, don't wait for all the requests to be executed before resubmitting a request in error.
[ https://issues.apache.org/jira/browse/HBASE-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-5924: --- Status: Patch Available (was: Open) In the client code, don't wait for all the requests to be executed before resubmitting a request in error. -- Key: HBASE-5924 URL: https://issues.apache.org/jira/browse/HBASE-5924 Project: HBase Issue Type: Improvement Components: client Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Fix For: 0.96.0 Attachments: 5924.v11.patch, 5924.v14.patch, 5924.v19.patch, 5924.v5.patch, 5924.v9.patch The client (in the function HConnectionManager#processBatchCallback) works in two steps: - make the requests - collect the failures and successes and prepare for retry It means that when there is an immediate error (region moved, split, dead server, ...) we still wait for all the initial requests to be executed before submitting again the failed request. If we have a scenario with all the requests taking 5 seconds we have a final execution time of: 5 (initial requests) + 1 (wait time) + 5 (final request) = 11s. We could improve this by analyzing immediately the results. This would lead us, for the scenario mentioned above, to 6 seconds. So we could have a performance improvement of nearly 50% in many cases, and much more than 50% if the request execution time is different. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5924) In the client code, don't wait for all the requests to be executed before resubmitting a request in error.
[ https://issues.apache.org/jira/browse/HBASE-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294612#comment-13294612 ] Hadoop QA commented on HBASE-5924: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12531976/5924.v19.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.TestDrainingServer Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2163//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2163//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2163//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2163//console This message is automatically generated. In the client code, don't wait for all the requests to be executed before resubmitting a request in error. -- Key: HBASE-5924 URL: https://issues.apache.org/jira/browse/HBASE-5924 Project: HBase Issue Type: Improvement Components: client Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Fix For: 0.96.0 Attachments: 5924.v11.patch, 5924.v14.patch, 5924.v19.patch, 5924.v5.patch, 5924.v9.patch The client (in the function HConnectionManager#processBatchCallback) works in two steps: - make the requests - collect the failures and successes and prepare for retry It means that when there is an immediate error (region moved, split, dead server, ...) we still wait for all the initial requests to be executed before submitting again the failed request. If we have a scenario with all the requests taking 5 seconds we have a final execution time of: 5 (initial requests) + 1 (wait time) + 5 (final request) = 11s. We could improve this by analyzing immediately the results. This would lead us, for the scenario mentioned above, to 6 seconds. So we could have a performance improvement of nearly 50% in many cases, and much more than 50% if the request execution time is different. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4391) Add ability to start RS as root and call mlockall
[ https://issues.apache.org/jira/browse/HBASE-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294622#comment-13294622 ] Matteo Bertozzi commented on HBASE-4391: Accumulo has a tserver.memory.lock property used to enable mlock in TabletServer. At startup the property is read and if true mlockall() is called. Different approach but the result is the same... the Todd proposed solution is external from the RegionServer code (and maybe can be integrated in hadoop-common to be used by datanode co without changing the hdfs code), the accumulo one is integrated in the tablet code... Add ability to start RS as root and call mlockall - Key: HBASE-4391 URL: https://issues.apache.org/jira/browse/HBASE-4391 Project: HBase Issue Type: New Feature Components: regionserver Affects Versions: 0.94.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.96.0 Attachments: HBASE-4391-v0.patch A common issue we've seen in practice is that users oversubscribe their region servers with too many MR tasks, etc. As soon as the machine starts swapping, the RS grinds to a halt, loses ZK session, aborts, etc. This can be combatted by starting the RS as root, calling mlockall(), and then setuid down to the hbase user. We should not require this, but we should provide it as an option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6028) Implement a cancel for in-progress compactions
[ https://issues.apache.org/jira/browse/HBASE-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294619#comment-13294619 ] Nicolas Spiegelberg commented on HBASE-6028: this should be trivial to implement. just remember that canceling a compaction also requires manually freeing the files under compaction. Implement a cancel for in-progress compactions -- Key: HBASE-6028 URL: https://issues.apache.org/jira/browse/HBASE-6028 Project: HBase Issue Type: Bug Components: regionserver Reporter: Derek Wollenstein Priority: Minor Labels: compaction, operations, regionserver Depending on current server load, it can be extremely expensive to run periodic minor / major compactions. It would be helpful to have a feature where a user could use the shell or a client tool to explicitly cancel an in-progress compactions. This would allow a system to recover when too many regions became eligible for compactions at once -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6188) Remove the concept of table owner
[ https://issues.apache.org/jira/browse/HBASE-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294583#comment-13294583 ] Laxman commented on HBASE-6188: --- Attached the new patch after fixing the review comments. This patch includes new testcases for owner and change of owner. Please review. Remove the concept of table owner - Key: HBASE-6188 URL: https://issues.apache.org/jira/browse/HBASE-6188 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.94.0, 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Laxman Labels: security Fix For: 0.96.0, 0.94.1 Attachments: HBASE-6188.1.patch, HBASE-6188.2.patch, HBASE-6188.patch The table owner concept was a design simplification in the initial drop. First, the design changes under review means only a user with GLOBAL CREATE permission can create a table, which will probably be an administrator. Then, granting implicit permissions may lead to oversights and it adds unnecessary conditionals to our code. So instead the administrator with GLOBAL CREATE permission should make the appropriate grants at table create time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6188) Remove the concept of table owner
[ https://issues.apache.org/jira/browse/HBASE-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laxman updated HBASE-6188: -- Attachment: HBASE-6188.2.patch Remove the concept of table owner - Key: HBASE-6188 URL: https://issues.apache.org/jira/browse/HBASE-6188 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.94.0, 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Laxman Labels: security Fix For: 0.96.0, 0.94.1 Attachments: HBASE-6188.1.patch, HBASE-6188.2.patch, HBASE-6188.patch The table owner concept was a design simplification in the initial drop. First, the design changes under review means only a user with GLOBAL CREATE permission can create a table, which will probably be an administrator. Then, granting implicit permissions may lead to oversights and it adds unnecessary conditionals to our code. So instead the administrator with GLOBAL CREATE permission should make the appropriate grants at table create time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6188) Remove the concept of table owner
[ https://issues.apache.org/jira/browse/HBASE-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laxman updated HBASE-6188: -- Status: Open (was: Patch Available) Remove the concept of table owner - Key: HBASE-6188 URL: https://issues.apache.org/jira/browse/HBASE-6188 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.94.0, 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Laxman Labels: security Fix For: 0.96.0, 0.94.1 Attachments: HBASE-6188.1.patch, HBASE-6188.2.patch, HBASE-6188.patch The table owner concept was a design simplification in the initial drop. First, the design changes under review means only a user with GLOBAL CREATE permission can create a table, which will probably be an administrator. Then, granting implicit permissions may lead to oversights and it adds unnecessary conditionals to our code. So instead the administrator with GLOBAL CREATE permission should make the appropriate grants at table create time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6188) Remove the concept of table owner
[ https://issues.apache.org/jira/browse/HBASE-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294630#comment-13294630 ] Hadoop QA commented on HBASE-6188: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532005/HBASE-6188.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2164//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2164//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2164//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2164//console This message is automatically generated. Remove the concept of table owner - Key: HBASE-6188 URL: https://issues.apache.org/jira/browse/HBASE-6188 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.94.0, 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Laxman Labels: security Fix For: 0.96.0, 0.94.1 Attachments: HBASE-6188.1.patch, HBASE-6188.2.patch, HBASE-6188.patch The table owner concept was a design simplification in the initial drop. First, the design changes under review means only a user with GLOBAL CREATE permission can create a table, which will probably be an administrator. Then, granting implicit permissions may lead to oversights and it adds unnecessary conditionals to our code. So instead the administrator with GLOBAL CREATE permission should make the appropriate grants at table create time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6192) Document ACL matrix in the book
[ https://issues.apache.org/jira/browse/HBASE-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laxman updated HBASE-6192: -- Attachment: HBase Security-ACL Matrix.xls HBase Security-ACL Matrix.pdf ACL Matrix attached in MS excel and pdf format. Feel free to modify if required. I'm not very sure where to include this matrix in hbase documentation. Document ACL matrix in the book --- Key: HBASE-6192 URL: https://issues.apache.org/jira/browse/HBASE-6192 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.96.0 Reporter: Enis Soztutar Assignee: Laxman Attachments: HBase Security-ACL Matrix.pdf, HBase Security-ACL Matrix.xls We have an excellent matrix at https://issues.apache.org/jira/secure/attachment/12531252/Security-ACL%20Matrix.pdf for ACL. Once the changes are done, we can adapt that and put it in the book, also add some more documentation about the new authorization features. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6143) Make region assignment smarter when regions are re-enabled.
[ https://issues.apache.org/jira/browse/HBASE-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294648#comment-13294648 ] Elliott Clark commented on HBASE-6143: -- I agree. Though sometimes choosing a balance of the regions can be an expensive operation so the balancer should provide an api for a cheap balance. Make region assignment smarter when regions are re-enabled. --- Key: HBASE-6143 URL: https://issues.apache.org/jira/browse/HBASE-6143 Project: HBase Issue Type: Improvement Reporter: Elliott Clark Assignee: Elliott Clark Right now a random region server is picked when re-enabling a table. This could be much smarter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6205) Support an option to keep data of dropped table for some time
[ https://issues.apache.org/jira/browse/HBASE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294664#comment-13294664 ] Andrew Purtell commented on HBASE-6205: --- +1 Support an option to keep data of dropped table for some time - Key: HBASE-6205 URL: https://issues.apache.org/jira/browse/HBASE-6205 Project: HBase Issue Type: New Feature Affects Versions: 0.94.0, 0.96.0 Reporter: chunhui shen Assignee: chunhui shen Attachments: HBASE-6205.patch User may drop table accidentally because of error code or other uncertain reasons. Unfortunately, it happens in our environment because one user make a mistake between production cluster and testing cluster. So, I just give a suggestion, do we need to support an option to keep data of dropped table for some time, eg.1 day. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6205) Support an option to keep data of dropped table for some time
[ https://issues.apache.org/jira/browse/HBASE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-6205: -- Component/s: (was: master) Affects Version/s: 0.96.0 0.94.0 Support an option to keep data of dropped table for some time - Key: HBASE-6205 URL: https://issues.apache.org/jira/browse/HBASE-6205 Project: HBase Issue Type: New Feature Affects Versions: 0.94.0, 0.96.0 Reporter: chunhui shen Assignee: chunhui shen Attachments: HBASE-6205.patch User may drop table accidentally because of error code or other uncertain reasons. Unfortunately, it happens in our environment because one user make a mistake between production cluster and testing cluster. So, I just give a suggestion, do we need to support an option to keep data of dropped table for some time, eg.1 day. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5713) Introduce throttling during Instant schema change process to throttle opening/closing regions.
[ https://issues.apache.org/jira/browse/HBASE-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294717#comment-13294717 ] Subbu M Iyer commented on HBASE-5713: - Ted: Attached a latest patch. 1. Reduced the throttle time to 100ms. 2. Addressed your comments. All unit tests passed. As a next step, we need to get this tested in a production grade cluster and hope Lars can help us with that. thanks Introduce throttling during Instant schema change process to throttle opening/closing regions. --- Key: HBASE-5713 URL: https://issues.apache.org/jira/browse/HBASE-5713 Project: HBase Issue Type: Bug Components: client, master, regionserver, shell Reporter: Subbu M Iyer Assignee: Subbu M Iyer Priority: Minor Attachments: 5713.txt, patch-v4.patch There is a potential for region open/close stampede during instant schema change process as the process attempts to close/open impacted regions in rapid succession. We need to introduce some kind of throttling to eliminate the race condition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5713) Introduce throttling during Instant schema change process to throttle opening/closing regions.
[ https://issues.apache.org/jira/browse/HBASE-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-5713: Attachment: patch-v4.patch Introduce throttling during Instant schema change process to throttle opening/closing regions. --- Key: HBASE-5713 URL: https://issues.apache.org/jira/browse/HBASE-5713 Project: HBase Issue Type: Bug Components: client, master, regionserver, shell Reporter: Subbu M Iyer Assignee: Subbu M Iyer Priority: Minor Attachments: 5713.txt, patch-v4.patch There is a potential for region open/close stampede during instant schema change process as the process attempts to close/open impacted regions in rapid succession. We need to introduce some kind of throttling to eliminate the race condition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294735#comment-13294735 ] Subbu M Iyer commented on HBASE-3909: - I have tried to capture all (most of) of configuration properties that are currently being used in an excel. The objective here is to figure out what properties may be effectively changed using the dynamic config utility and what is not. While doing this exercise I figured that almost close to 200 properties are missing from hbase-default.xml but is getting used in the system. So, here are my thoughts on this and would like to hear from you guys your thoughts as well: 1. My understanding is that all configuration properties will be/must be defined in hbase-default.xml with a default value. Some exceptions are possible in some corner cases but majority of properties must be defined here. 2. There is no way for a user to know that a property is configurable/changeable unless 1) the property is defined in hbase-default 2) Or communicated through some documentation or API doc. Add dynamic config -- Key: HBASE-3909 URL: https://issues.apache.org/jira/browse/HBASE-3909 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.96.0 Attachments: 3909-v1.patch, 3909.v1 I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue. Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294736#comment-13294736 ] Subbu M Iyer commented on HBASE-3909: - Ted/Uma: Addressed all of your comments in this patch. Add dynamic config -- Key: HBASE-3909 URL: https://issues.apache.org/jira/browse/HBASE-3909 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.96.0 Attachments: 3909-v1.patch, 3909.v1, HBase Cluster Config Details.xlsx, patch-v2.patch I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue. Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-3909: Attachment: HBase Cluster Config Details.xlsx Add dynamic config -- Key: HBASE-3909 URL: https://issues.apache.org/jira/browse/HBASE-3909 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.96.0 Attachments: 3909-v1.patch, 3909.v1, HBase Cluster Config Details.xlsx, patch-v2.patch I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue. Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-3909: Attachment: patch-v2.patch Add dynamic config -- Key: HBASE-3909 URL: https://issues.apache.org/jira/browse/HBASE-3909 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.96.0 Attachments: 3909-v1.patch, 3909.v1, HBase Cluster Config Details.xlsx, patch-v2.patch I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue. Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294740#comment-13294740 ] Subbu M Iyer commented on HBASE-3909: - I am seeing that this is a complicated feature with lots of open questions. 1. There are lot of properties/configuration that may have no effect (even with dynamic config tool) once the server is up. How do we communicate such cases? Do we maintain a smaller subset of configurations that can be changed dynamically and only allow those subsets for dynamic changes? 2. There are lot of properties that might take effect during cases of server fail over/back up server taking over etc. Basically every dynamic configuration change that we apply to the cluster will take effect during server failover/ or back server taking over. How do we communicate such cases? Do we even want this? 3. Do we really think that this feature will be a valuable addition? Add dynamic config -- Key: HBASE-3909 URL: https://issues.apache.org/jira/browse/HBASE-3909 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.96.0 Attachments: 3909-v1.patch, 3909.v1, HBase Cluster Config Details.xlsx, patch-v2.patch I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue. Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6184) HRegionInfo was null or empty in Meta
[ https://issues.apache.org/jira/browse/HBASE-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiafeng.zhang updated HBASE-6184: - Attachment: (was: HBASE-6184.patch) HRegionInfo was null or empty in Meta -- Key: HBASE-6184 URL: https://issues.apache.org/jira/browse/HBASE-6184 Project: HBase Issue Type: Bug Components: client, io Affects Versions: 0.94.0 Reporter: jiafeng.zhang Fix For: 0.94.0 insert data hadoop-0.23.2 + hbase-0.94.0 2012-06-07 13:09:38,573 WARN [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation] Encountered problems when prefetch META table: java.io.IOException: HRegionInfo was null or empty in Meta for hbase_one_col, row=hbase_one_col,09115303780247449149,99 at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:160) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1482) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1367) at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:945) at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:801) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:776) at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:397) at com.dinglicom.hbase.HbaseImport.insertData(HbaseImport.java:177) at com.dinglicom.hbase.HbaseImport.run(HbaseImport.java:210) at java.lang.Thread.run(Thread.java:662) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6202) Medium tests fail with jdk1.7
[ https://issues.apache.org/jira/browse/HBASE-6202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-6202: --- Status: Patch Available (was: Open) Fixed the failed tests. Medium tests fail with jdk1.7 - Key: HBASE-6202 URL: https://issues.apache.org/jira/browse/HBASE-6202 Project: HBase Issue Type: Sub-task Components: test Affects Versions: 0.96.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: 6202-hbase.patch Failed tests: testOrphanTaskAcquisition(org.apache.hadoop.hbase.master.TestSplitLogManager) testCreateAndUpdate(org.apache.hadoop.hbase.util.TestFSTableDescriptors): statuses.length=5 testManageSingleton(org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager) Tests in error: testRegionTransitionOperations(org.apache.hadoop.hbase.coprocessor.TestMasterObserver): org.apache.hadoop.hbase.UnknownRegionException: ef07dae0851f4fbe7d550413530b8774 testTableOperations(org.apache.hadoop.hbase.coprocessor.TestMasterObserver): org.apache.hadoop.hbase.TableExistsException: observed_table testClassLoadingFromHDFS(org.apache.hadoop.hbase.coprocessor.TestClassLoading): org.apache.hadoop.hbase.TableNotEnabledException: TestClassLoading -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6202) Medium tests fail with jdk1.7
[ https://issues.apache.org/jira/browse/HBASE-6202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-6202: --- Attachment: 6202-hbase.patch Medium tests fail with jdk1.7 - Key: HBASE-6202 URL: https://issues.apache.org/jira/browse/HBASE-6202 Project: HBase Issue Type: Sub-task Components: test Affects Versions: 0.96.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: 6202-hbase.patch Failed tests: testOrphanTaskAcquisition(org.apache.hadoop.hbase.master.TestSplitLogManager) testCreateAndUpdate(org.apache.hadoop.hbase.util.TestFSTableDescriptors): statuses.length=5 testManageSingleton(org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager) Tests in error: testRegionTransitionOperations(org.apache.hadoop.hbase.coprocessor.TestMasterObserver): org.apache.hadoop.hbase.UnknownRegionException: ef07dae0851f4fbe7d550413530b8774 testTableOperations(org.apache.hadoop.hbase.coprocessor.TestMasterObserver): org.apache.hadoop.hbase.TableExistsException: observed_table testClassLoadingFromHDFS(org.apache.hadoop.hbase.coprocessor.TestClassLoading): org.apache.hadoop.hbase.TableNotEnabledException: TestClassLoading -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6206) Large tests fail with jdk1.7
Jimmy Xiang created HBASE-6206: -- Summary: Large tests fail with jdk1.7 Key: HBASE-6206 URL: https://issues.apache.org/jira/browse/HBASE-6206 Project: HBase Issue Type: Sub-task Components: thrift Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Failed tests: testSimplePutDelete(org.apache.hadoop.hbase.replication.TestMasterReplication): Waited too much time for put replication testCyclicReplication(org.apache.hadoop.hbase.replication.TestMasterReplication): Waited too much time for put replication testMultiSlaveReplication(org.apache.hadoop.hbase.replication.TestMultiSlaveReplication): Waited too much time for put replication testDeleteTypes(org.apache.hadoop.hbase.replication.TestReplication): Waited too much time for put replication testSimplePutDelete(org.apache.hadoop.hbase.replication.TestReplication): Waited too much time for put replication testSmallBatch(org.apache.hadoop.hbase.replication.TestReplication): Waited too much time for normal batch replication testStartStop(org.apache.hadoop.hbase.replication.TestReplication): Waited too much time for put replication testDisableEnable(org.apache.hadoop.hbase.replication.TestReplication): Waited too much time for put replication testDisableInactivePeer(org.apache.hadoop.hbase.replication.TestReplication): Waited too much time for put replication testAddAndRemoveClusters(org.apache.hadoop.hbase.replication.TestReplication): Waited too much time for put replication loadTesting(org.apache.hadoop.hbase.replication.TestReplication): Waited too much time for normal batch replication, 0 instead of 1000 testVerifyRepJob(org.apache.hadoop.hbase.replication.TestReplication): Waited too much time for normal batch replication testRSSplitEphemeralsDisappearButDaughtersAreOnlinedAfterShutdownHandling(org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster) Tests in error: testNull(org.apache.hadoop.hbase.client.TestFromClientSide) testPut(org.apache.hadoop.hbase.client.TestFromClientSide) testSplit(org.apache.hadoop.hbase.regionserver.wal.TestHLog): 3 exceptions [org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /user/jxiang/hbase/TestHLog/08fec3b92db148f74a1599c3db6fb1ee/recovered.edits/004.temp File is not open for writing. Holder DFSClient_388157531 does not have any open files.(..) Tests run: 333, Failures: 4, Errors: 3, Skipped: 7 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6184) HRegionInfo was null or empty in Meta
[ https://issues.apache.org/jira/browse/HBASE-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiafeng.zhang updated HBASE-6184: - Attachment: HBASE-6184.patch HRegionInfo was null or empty in Meta -- Key: HBASE-6184 URL: https://issues.apache.org/jira/browse/HBASE-6184 Project: HBase Issue Type: Bug Components: client, io Affects Versions: 0.94.0 Reporter: jiafeng.zhang Fix For: 0.94.0 Attachments: HBASE-6184.patch insert data hadoop-0.23.2 + hbase-0.94.0 2012-06-07 13:09:38,573 WARN [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation] Encountered problems when prefetch META table: java.io.IOException: HRegionInfo was null or empty in Meta for hbase_one_col, row=hbase_one_col,09115303780247449149,99 at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:160) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1482) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1367) at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:945) at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:801) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:776) at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:397) at com.dinglicom.hbase.HbaseImport.insertData(HbaseImport.java:177) at com.dinglicom.hbase.HbaseImport.run(HbaseImport.java:210) at java.lang.Thread.run(Thread.java:662) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6205) Support an option to keep data of dropped table for some time
[ https://issues.apache.org/jira/browse/HBASE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-6205: Attachment: HBASE-6205v2.patch Updating patch v2 Support an option to keep data of dropped table for some time - Key: HBASE-6205 URL: https://issues.apache.org/jira/browse/HBASE-6205 Project: HBase Issue Type: New Feature Affects Versions: 0.94.0, 0.96.0 Reporter: chunhui shen Assignee: chunhui shen Attachments: HBASE-6205.patch, HBASE-6205v2.patch User may drop table accidentally because of error code or other uncertain reasons. Unfortunately, it happens in our environment because one user make a mistake between production cluster and testing cluster. So, I just give a suggestion, do we need to support an option to keep data of dropped table for some time, eg.1 day. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6205) Support an option to keep data of dropped table for some time
[ https://issues.apache.org/jira/browse/HBASE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-6205: Description: User may drop table accidentally because of error code or other uncertain reasons. Unfortunately, it happens in our environment because one user make a mistake between production cluster and testing cluster. So, I just give a suggestion, do we need to support an option to keep data of dropped table for some time, e.g. 1 day In the patch: We make a new dir named .trashtables in the rood dir. In the DeleteTableHandler, we move files in dropped table's dir to trash table dir instead of deleting them directly. And Create new class TrashTableCleaner which will clean dropped tables if it is time out with a period check. Default keep time for dropped tables is 1 day, and check period is 1 hour. was: User may drop table accidentally because of error code or other uncertain reasons. Unfortunately, it happens in our environment because one user make a mistake between production cluster and testing cluster. So, I just give a suggestion, do we need to support an option to keep data of dropped table for some time, eg.1 day. Support an option to keep data of dropped table for some time - Key: HBASE-6205 URL: https://issues.apache.org/jira/browse/HBASE-6205 Project: HBase Issue Type: New Feature Affects Versions: 0.94.0, 0.96.0 Reporter: chunhui shen Assignee: chunhui shen Attachments: HBASE-6205.patch, HBASE-6205v2.patch User may drop table accidentally because of error code or other uncertain reasons. Unfortunately, it happens in our environment because one user make a mistake between production cluster and testing cluster. So, I just give a suggestion, do we need to support an option to keep data of dropped table for some time, e.g. 1 day In the patch: We make a new dir named .trashtables in the rood dir. In the DeleteTableHandler, we move files in dropped table's dir to trash table dir instead of deleting them directly. And Create new class TrashTableCleaner which will clean dropped tables if it is time out with a period check. Default keep time for dropped tables is 1 day, and check period is 1 hour. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6184) HRegionInfo was null or empty in Meta
[ https://issues.apache.org/jira/browse/HBASE-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294780#comment-13294780 ] Jimmy Xiang commented on HBASE-6184: How will this fix the issue? Have you checked the data in the meta table? Does that row have region info? I think this most likely is a meta table data issue. HRegionInfo was null or empty in Meta -- Key: HBASE-6184 URL: https://issues.apache.org/jira/browse/HBASE-6184 Project: HBase Issue Type: Bug Components: client, io Affects Versions: 0.94.0 Reporter: jiafeng.zhang Fix For: 0.94.0 Attachments: HBASE-6184.patch insert data hadoop-0.23.2 + hbase-0.94.0 2012-06-07 13:09:38,573 WARN [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation] Encountered problems when prefetch META table: java.io.IOException: HRegionInfo was null or empty in Meta for hbase_one_col, row=hbase_one_col,09115303780247449149,99 at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:160) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1482) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1367) at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:945) at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:801) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:776) at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:397) at com.dinglicom.hbase.HbaseImport.insertData(HbaseImport.java:177) at com.dinglicom.hbase.HbaseImport.run(HbaseImport.java:210) at java.lang.Thread.run(Thread.java:662) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6202) Medium tests fail with jdk1.7
[ https://issues.apache.org/jira/browse/HBASE-6202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294784#comment-13294784 ] Hadoop QA commented on HBASE-6202: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532032/6202-hbase.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.coprocessor.TestClassLoading org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2165//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2165//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2165//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2165//console This message is automatically generated. Medium tests fail with jdk1.7 - Key: HBASE-6202 URL: https://issues.apache.org/jira/browse/HBASE-6202 Project: HBase Issue Type: Sub-task Components: test Affects Versions: 0.96.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: 6202-hbase.patch Failed tests: testOrphanTaskAcquisition(org.apache.hadoop.hbase.master.TestSplitLogManager) testCreateAndUpdate(org.apache.hadoop.hbase.util.TestFSTableDescriptors): statuses.length=5 testManageSingleton(org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager) Tests in error: testRegionTransitionOperations(org.apache.hadoop.hbase.coprocessor.TestMasterObserver): org.apache.hadoop.hbase.UnknownRegionException: ef07dae0851f4fbe7d550413530b8774 testTableOperations(org.apache.hadoop.hbase.coprocessor.TestMasterObserver): org.apache.hadoop.hbase.TableExistsException: observed_table testClassLoadingFromHDFS(org.apache.hadoop.hbase.coprocessor.TestClassLoading): org.apache.hadoop.hbase.TableNotEnabledException: TestClassLoading -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6207) Add jitter to client retry timer
Elliott Clark created HBASE-6207: Summary: Add jitter to client retry timer Key: HBASE-6207 URL: https://issues.apache.org/jira/browse/HBASE-6207 Project: HBase Issue Type: Improvement Reporter: Elliott Clark Assignee: Elliott Clark -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6162) Move KeyValue to hbase-common module
[ https://issues.apache.org/jira/browse/HBASE-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Corgan updated HBASE-6162: --- Attachment: HBASE-6162-v3.patch Maybe we want to avoid messing with the tests for now. Here is a v3 patch that does the minimum possible to move KeyValue to hbase-common. It brings HeapSize and ClassSize with it, and it moves a couple constants around. After this jira I'll move the DataBlockEncoding base classes and then add the prefix-trie module. Move KeyValue to hbase-common module Key: HBASE-6162 URL: https://issues.apache.org/jira/browse/HBASE-6162 Project: HBase Issue Type: Improvement Affects Versions: 0.96.0 Reporter: Matt Corgan Assignee: Matt Corgan Fix For: 0.96.0 Attachments: HBASE-6162-v1.patch, HBASE-6162-v2.patch, HBASE-6162-v3.patch * pull KeyValue up to hbase-common module This is part of the modularization strategy in HBASE-5977, and is specifically necessary to modularize HBASE-4676. also brings these classes to hbase-common: * ClassSize, HeapSize * HTestConst * TestKeyValue, KeyValueTestUtil * LoadTestKVGenerator, TestLoadTestKVGenerator * MD5Hash moves a trivial constant (HRegionInfo.DELIMITER) from HRegionInfo to HConstants -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6192) Document ACL matrix in the book
[ https://issues.apache.org/jira/browse/HBASE-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laxman updated HBASE-6192: -- Component/s: documentation Affects Version/s: 0.94.1 Tags: documentation, hbase book Labels: documentaion security (was: ) Document ACL matrix in the book --- Key: HBASE-6192 URL: https://issues.apache.org/jira/browse/HBASE-6192 Project: HBase Issue Type: Sub-task Components: documentation, security Affects Versions: 0.96.0, 0.94.1 Reporter: Enis Soztutar Assignee: Laxman Labels: documentaion, security Attachments: HBase Security-ACL Matrix.pdf, HBase Security-ACL Matrix.xls We have an excellent matrix at https://issues.apache.org/jira/secure/attachment/12531252/Security-ACL%20Matrix.pdf for ACL. Once the changes are done, we can adapt that and put it in the book, also add some more documentation about the new authorization features. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5974) Scanner retry behavior with RPC timeout on next() seems incorrect
[ https://issues.apache.org/jira/browse/HBASE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294812#comment-13294812 ] Anoop Sam John commented on HBASE-5974: --- 2 test classes need change TestAssignmentManager#processServerShutdownHandler() TestMetaReaderEditorNoCluster#testRideOverServerNotRunning () I will give a new patch with these test case changes needed as per the patch soon. Scanner retry behavior with RPC timeout on next() seems incorrect - Key: HBASE-5974 URL: https://issues.apache.org/jira/browse/HBASE-5974 Project: HBase Issue Type: Bug Components: client, regionserver Affects Versions: 0.90.7, 0.92.1, 0.94.0, 0.96.0 Reporter: Todd Lipcon Assignee: Anoop Sam John Priority: Critical Fix For: 0.96.0, 0.94.1 Attachments: 5974_94-V4.patch, 5974_trunk-V2.patch, 5974_trunk.patch, HBASE-5974_0.94.patch, HBASE-5974_94-V2.patch, HBASE-5974_94-V3.patch I'm seeing the following behavior: - set RPC timeout to a short value - call next() for some batch of rows, big enough so the client times out before the result is returned - the HConnectionManager stuff will retry the next() call to the same server. At this point, one of two things can happen: 1) the previous next() call will still be processing, in which case you get a LeaseException, because it was removed from the map during the processing, or 2) the next() call will succeed but skip the prior batch of rows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6205) Support an option to keep data of dropped table for some time
[ https://issues.apache.org/jira/browse/HBASE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6205: -- Fix Version/s: 0.96.0 Status: Patch Available (was: Open) Support an option to keep data of dropped table for some time - Key: HBASE-6205 URL: https://issues.apache.org/jira/browse/HBASE-6205 Project: HBase Issue Type: New Feature Affects Versions: 0.94.0, 0.96.0 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6205.patch, HBASE-6205v2.patch User may drop table accidentally because of error code or other uncertain reasons. Unfortunately, it happens in our environment because one user make a mistake between production cluster and testing cluster. So, I just give a suggestion, do we need to support an option to keep data of dropped table for some time, e.g. 1 day In the patch: We make a new dir named .trashtables in the rood dir. In the DeleteTableHandler, we move files in dropped table's dir to trash table dir instead of deleting them directly. And Create new class TrashTableCleaner which will clean dropped tables if it is time out with a period check. Default keep time for dropped tables is 1 day, and check period is 1 hour. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6205) Support an option to keep data of dropped table for some time
[ https://issues.apache.org/jira/browse/HBASE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294817#comment-13294817 ] Ted Yu commented on HBASE-6205: --- Nice feature. Shall we consider the possibility that there may be more than one attempt to delete the same table ? This could be due to e.g. region server failure, etc. In that case we would have more than one directory for the table, each with different timestamp prefix. It would be nice for this feature to group regions for the same table under one parent directory. Further, trash is able to hold more than one tables. Can we name related classes Trash instead of TrashTable ? The regions in trash would occupy disk space. It would be nice to show on master UI how much space is currently consumed by trash, what tables are in trash. It would also be nice to prompt user if he/she is creating a table with the same name as one of the tables in trash. The above can be done in follow-up JIRAs. However we should prepare for enhancement from the beginning. Please put patch on review board when you think it has paved the way for future enhancement. Support an option to keep data of dropped table for some time - Key: HBASE-6205 URL: https://issues.apache.org/jira/browse/HBASE-6205 Project: HBase Issue Type: New Feature Affects Versions: 0.94.0, 0.96.0 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6205.patch, HBASE-6205v2.patch User may drop table accidentally because of error code or other uncertain reasons. Unfortunately, it happens in our environment because one user make a mistake between production cluster and testing cluster. So, I just give a suggestion, do we need to support an option to keep data of dropped table for some time, e.g. 1 day In the patch: We make a new dir named .trashtables in the rood dir. In the DeleteTableHandler, we move files in dropped table's dir to trash table dir instead of deleting them directly. And Create new class TrashTableCleaner which will clean dropped tables if it is time out with a period check. Default keep time for dropped tables is 1 day, and check period is 1 hour. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6162) Move KeyValue to hbase-common module
[ https://issues.apache.org/jira/browse/HBASE-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13294820#comment-13294820 ] Ted Yu commented on HBASE-6162: --- From https://builds.apache.org/job/PreCommit-HBASE-Build/2166/console, looks like there might be compilation issue. Move KeyValue to hbase-common module Key: HBASE-6162 URL: https://issues.apache.org/jira/browse/HBASE-6162 Project: HBase Issue Type: Improvement Affects Versions: 0.96.0 Reporter: Matt Corgan Assignee: Matt Corgan Fix For: 0.96.0 Attachments: HBASE-6162-v1.patch, HBASE-6162-v2.patch, HBASE-6162-v3.patch * pull KeyValue up to hbase-common module This is part of the modularization strategy in HBASE-5977, and is specifically necessary to modularize HBASE-4676. also brings these classes to hbase-common: * ClassSize, HeapSize * HTestConst * TestKeyValue, KeyValueTestUtil * LoadTestKVGenerator, TestLoadTestKVGenerator * MD5Hash moves a trivial constant (HRegionInfo.DELIMITER) from HRegionInfo to HConstants -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
