[jira] [Commented] (HBASE-4903) Return a result from RegionObserver.preIncrement()
[ https://issues.apache.org/jira/browse/HBASE-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161474#comment-13161474 ] Hudson commented on HBASE-4903: --- Integrated in HBase-0.92 #168 (See [https://builds.apache.org/job/HBase-0.92/168/]) HBASE-4903 addendum for security build larsh : Files : * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java Return a result from RegionObserver.preIncrement() -- Key: HBASE-4903 URL: https://issues.apache.org/jira/browse/HBASE-4903 Project: HBase Issue Type: Improvement Reporter: Daniel Gómez Ferro Fix For: 0.94.0, 0.92.1 Attachments: 4903-add.txt, HBASE-4903-0.92.patch, HBASE-4903.patch, HBASE-4903.patch The only way to return a result from RegionObserver.preIncrement() is to use Result.readFields() after serializing the correct result. This can be fixed either returning a Result object from that function or adding setters to Result. Another option is to modify the parameters and receive a ListKeyValue as preGet() does. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4899) Region would be assigned twice easily with continually killing server and moving region in testing environment
[
https://issues.apache.org/jira/browse/HBASE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161473#comment-13161473
]
Hudson commented on HBASE-4899:
---
Integrated in HBase-0.92 #168 (See
[https://builds.apache.org/job/HBase-0.92/168/])
HBASE-4899 Region would be assigned twice easily with continually killing
server and moving region in testing environment
stack :
Files :
* /hbase/branches/0.92/CHANGES.txt
*
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java
Region would be assigned twice easily with continually killing server and
moving region in testing environment
---
Key: HBASE-4899
URL: https://issues.apache.org/jira/browse/HBASE-4899
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.92.1
Attachments: hbase-4899.patch, hbase-4899v2.patch, hbase-4899v3.patch
Before assigning region in ServerShutdownHandler#process, it will check
whether region is in RIT,
however, this checking doesn't work as the excepted in the following case:
1.move region A from server B to server C
2.kill server B
3.start server B immediately
Let's see what happen in the code for the above case
{code}
for step1:
1.1 server B close the region A,
1.2 master setOffline for region
A,(AssignmentManager#setOffline:this.regions.remove(regionInfo))
1.3 server C start to open region A.(Not completed)
for step3:
master ServerShutdownHandler#process() for server B
{
..
splitlog()
...
ListRegionState regionsInTransition =
this.services.getAssignmentManager()
.processServerShutdown(this.serverName);
...
Skip regions that were in transition unless CLOSING or PENDING_CLOSE
...
assign region
}
{code}
In fact, when running
ServerShutdownHandler#process()#this.services.getAssignmentManager().processServerShutdown(this.serverName),
region A is in RIT (step1.3 not completed), but the return ListRegionState
regionsInTransition doesn't contain it, because region A has removed from
AssignmentManager.regions by AssignmentManager#setOffline in step 1.2
Therefore, region A will be assigned twice.
Actually, one server killed and started twice will also easily cause region
assigned twice.
Exclude the above reason, another probability :
when execute ServerShutdownHandler#process()#MetaReader.getServerUserRegions
,region is included which is in RIT now.
But after completing MetaReader.getServerUserRegions, the region has been
opened in other server and is not in RIT now.
In our testing environment where balancing,moving and killing are executed
periodly, assigning region twice often happens, and it is hateful because it
will affect other test cases.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4933) Ability to calculate the blockcache hit ratio for the last few minutes
[ https://issues.apache.org/jira/browse/HBASE-4933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161479#comment-13161479 ] dhruba borthakur commented on HBASE-4933: - https://reviews.facebook.net/D585 Ability to calculate the blockcache hit ratio for the last few minutes -- Key: HBASE-4933 URL: https://issues.apache.org/jira/browse/HBASE-4933 Project: HBase Issue Type: Improvement Components: metrics Reporter: dhruba borthakur Assignee: dhruba borthakur The metric blockcacheHitRatio is since the beginning of time. It would be nice to calculate the block cache hit ratio for the past few minutes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161498#comment-13161498 ] nkeywal commented on HBASE-4712: bq. Is that one report each for the medium and large or one report for the two? Just need a single word of clarification in there One report for the two. I will reformulate this part. bq. Is this new behavior from what's in trunk currently? Because I'm pretty sure I can just run mvn test -Dtest=myTest and get it to run them. Yes, it would change from what we have in trunk today. This profile deactivates categories, if not re they are still taken into account even if you specify -Dtest=myTest, so if your test is in largeTests it's not executed. bq. I don't think we really want to have these options available, right? A lot of the large tests depend on the fact that jvms are forked. My bad, copy/paste error, I am going to fix it bq. A simple example or two here with the annotations would make this super obvious. For example: Ok, will add an example. bq. Do we need to add a MiniRegion just for testing these situations? It runs like that out of the box (the region can use the local system), see for example TestScanner. bq. It would be cool if we could actually do that with the annotation. For instance: @LargeTest(modifications=false) If we can't check automatically that the annotation does not lie, we have the risk of having to look at the source code to check if it's true. As well, we can have test with a default cluster but that cannot run with other tests (if they try to timout the master for example), so I believe we will have to be explicit (something like sharedCluster=true). Note as well that a test can belong to multiple categories. Thanks for the review! Document rules for writing tests Key: HBASE-4712 URL: https://issues.apache.org/jira/browse/HBASE-4712 Project: HBase Issue Type: Task Components: test Affects Versions: 0.92.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor We saw that some tests could be improved. Documenting the general rules could help. Proposal: HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster. Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally. Small and medium tests must not need more than 30 minutes to run altogether. Small and medium tests should be executed by the developers before submitting a patch. Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests. Tests rules hints are: - As most as possible, tests should be written as small tests. - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names. - All tests should be written to be as fast as possible. - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests. - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster. - Sleeps: - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances. - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop. - Tests using cluster: - Tests using a HRegion do not have to start a cluster: A region can use the local file system. - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class. - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories. - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
[
https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161499#comment-13161499
]
nkeywal commented on HBASE-4712:
updated version after Jesse's comments:
--
1) Running tests
HBase tests are divided in three categories: small, medium and large, with
corresponding JUnit categories: SmallTests, MediumTests, LargeTests.
This appears like this:
{noformat}
@Category(SmallTests.class)
public class TestHRegionInfo {
@Test
public void testCreateHRegionInfoName() throws Exception {
// ...
}
}
{noformat}
Small tests are executed in a shared JVM. We put in this category all the tests
that can be executed quickly (the maximum execution time for a test is 15
seconds, and they do not use a cluster) in a shared jvm.
Medium tests represents tests that must be executed before proposing a patch.
They are designed to run in less than 30 minutes altogether, and are quite
stable in their results. They're designed to last less than 50 seconds
individually. They can use a cluster, and each of them is executed in a
separate JVM.
Large tests are everything else. They are typically integration tests,
regression tests for specific bugs, timeout tests, performance tests. Some of
them can be flaky. They are executed before a commit on the pre-integration
machines. They can be run on the developer machine as well.
Commands are:
1.1) mvn test
- execute small tests in a single JVM and medium tests in a separate JVM for
each test
- medium tests are NOT executed if there is an error in a small test
- large tests are NOT executed
- there are two reports, one report for small tests, and one report for medium
tests if they are executed
1.2) mvn test -P runAllTests
- execute small tests in a single JVM then medium and large tests in a separate
JVM for each test.
- medium and large tests are NOT executed if there is an error in a small test
- there are two reports, one report for small tests, and one report for medium
and large tests if they are executed
1.3) mvn test -P localTests -Dtest=myTests
- remove any category effect (without this specific profile, the profiles are
taken into account)
- use actually the official release of surefire the old connector to junit
- tests are executed in separated JVM
- you will see a new message at the end of the report: [INFO] Tests are
skipped. It's harmless.
1.4) Various other profiles
- mvn test -P runSmallTests - execute small tests only, in a single JVM.
- mvn test -P runMediumTests - execute medium tests in a separate JVM.
- mvn test -P runLargeTests - execute large tests in a separate JVM.
It's as well possible to use the script 'hbasetests.sh'. This script runs the
medium and large tests in parallel with two maven instances, and provide a
single report. It must be executed from the directory which contains the
pom.xml. Commands are:
./dev-support/hbasetests.sh - execute small and medium tests
./dev-support/hbasetests.sh runAllTests - execute all tests
./dev-support/hbasetests.sh replayFailed - rerun the failed tests a second
time, in a separate jvm and without parallelisation.
2) Writing tests
Tests rules hints are:
- As most as possible, tests should be written as small tests.
- All tests must be written to support parallel execution on the same machine,
hence should not use shared resources as fixed ports or fixed file names.
- Tests should not overlog. More than 100 lines/second makes the logs complex
to read and use i/o that are hence not available for the other tests.
- Tests can be written with HBaseTestingUtility . This class offers helper
functions to create a temp directory and do the cleanup, or to start a cluster.
Categories and execution time
- All tests must be categorized, if not they could be skipped.
- All tests should be written to be as fast as possible.
- Small tests should last less than 15 seconds, and must not have any side
effect.
- Medium tests should last less than 45 seconds.
- large tests should last less than 3 minutes, this ensure a good
parallelization for people using it, and ease the analysis when the test fails.
Sleeps
- Whenever possible, tests should not use Thread.sleep, but rather waiting for
the real event they need. This is faster and clearer for the reader.
- Tests should not do a 'Thread.sleep' without testing an ending condition.
This allows understanding what the test is waiting for. Moreover, the test will
work whatever the machine performances.
- Sleep should be minimal to be as fast as possible. Waiting for a variable
should be done in a 40ms sleep loop. Waiting for a socket operation should be
done in a 200 ms sleep loop.
Tests using a cluster
- Tests using a HRegion do not have to start a cluster: A region can use the
local file system instead of a distributed one.
- Start/stopping a cluster cost around 10 seconds. They should not be started
per test method but per test
[jira] [Commented] (HBASE-4880) Region is on service before completing openRegionHanlder, may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161502#comment-13161502
]
chunhui shen commented on HBASE-4880:
-
How to deal with this issue?
Region is on service before completing openRegionHanlder, may cause data loss
-
Key: HBASE-4880
URL: https://issues.apache.org/jira/browse/HBASE-4880
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: hbase-4880.patch
OpenRegionHandler in regionserver is processed as the following steps:
{code}
1.openregion()(Through it, closed = false, closing = false)
2.addToOnlineRegions(region)
3.update .meta. table
4.update ZK's node state to RS_ZK_REGION_OPEND
{code}
We can find that region is on service before Step 4.
It means client could put data to this region after step 3.
What will happen if step 4 is failed processing?
It will execute OpenRegionHandler#cleanupFailedOpen which will do closing
region, and master assign this region to another regionserver.
If closing region is failed, the data which is put between step 3 and step 4
may loss, because the region has been opend on another regionserver and be
put new data. Therefore, it may not be recoverd through replayRecoveredEdit()
because the edit's LogSeqId is smaller than current region SeqId.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4903) Return a result from RegionObserver.preIncrement()
[ https://issues.apache.org/jira/browse/HBASE-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161515#comment-13161515 ] Hudson commented on HBASE-4903: --- Integrated in HBase-TRUNK #2508 (See [https://builds.apache.org/job/HBase-TRUNK/2508/]) HBASE-4903 addendum for security build larsh : Files : * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java Return a result from RegionObserver.preIncrement() -- Key: HBASE-4903 URL: https://issues.apache.org/jira/browse/HBASE-4903 Project: HBase Issue Type: Improvement Reporter: Daniel Gómez Ferro Fix For: 0.94.0, 0.92.1 Attachments: 4903-add.txt, HBASE-4903-0.92.patch, HBASE-4903.patch, HBASE-4903.patch The only way to return a result from RegionObserver.preIncrement() is to use Result.readFields() after serializing the correct result. This can be fixed either returning a Result object from that function or adding setters to Result. Another option is to modify the parameters and receive a ListKeyValue as preGet() does. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4899) Region would be assigned twice easily with continually killing server and moving region in testing environment
[
https://issues.apache.org/jira/browse/HBASE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161514#comment-13161514
]
Hudson commented on HBASE-4899:
---
Integrated in HBase-TRUNK #2508 (See
[https://builds.apache.org/job/HBase-TRUNK/2508/])
HBASE-4899 Region would be assigned twice easily with continually killing
server and moving region in testing environment
stack :
Files :
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java
Region would be assigned twice easily with continually killing server and
moving region in testing environment
---
Key: HBASE-4899
URL: https://issues.apache.org/jira/browse/HBASE-4899
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.92.1
Attachments: hbase-4899.patch, hbase-4899v2.patch, hbase-4899v3.patch
Before assigning region in ServerShutdownHandler#process, it will check
whether region is in RIT,
however, this checking doesn't work as the excepted in the following case:
1.move region A from server B to server C
2.kill server B
3.start server B immediately
Let's see what happen in the code for the above case
{code}
for step1:
1.1 server B close the region A,
1.2 master setOffline for region
A,(AssignmentManager#setOffline:this.regions.remove(regionInfo))
1.3 server C start to open region A.(Not completed)
for step3:
master ServerShutdownHandler#process() for server B
{
..
splitlog()
...
ListRegionState regionsInTransition =
this.services.getAssignmentManager()
.processServerShutdown(this.serverName);
...
Skip regions that were in transition unless CLOSING or PENDING_CLOSE
...
assign region
}
{code}
In fact, when running
ServerShutdownHandler#process()#this.services.getAssignmentManager().processServerShutdown(this.serverName),
region A is in RIT (step1.3 not completed), but the return ListRegionState
regionsInTransition doesn't contain it, because region A has removed from
AssignmentManager.regions by AssignmentManager#setOffline in step 1.2
Therefore, region A will be assigned twice.
Actually, one server killed and started twice will also easily cause region
assigned twice.
Exclude the above reason, another probability :
when execute ServerShutdownHandler#process()#MetaReader.getServerUserRegions
,region is included which is in RIT now.
But after completing MetaReader.getServerUserRegions, the region has been
opened in other server and is not in RIT now.
In our testing environment where balancing,moving and killing are executed
periodly, assigning region twice often happens, and it is hateful because it
will affect other test cases.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from 0.89-fb)
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-4908: --- Attachment: D549.3.patch mbautin updated the revision [jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb). Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA Debugged and significantly improved RestartMetaTest and its dependency ProcessBasedLocalHBaseCluster to the point of actually being able to run a local multi-process test of an HBase cluster that kills the region server with the .META. region and splits the logs. Discovered that our log splitting does not work correctly with RawLocalFileSystem or checksum-enabled local filesystem, and added a fix to the hack in SequenceFileLogReader to only apply reflection logic to DFSInputStream. Still need to clean up the code, re-run unit tests, and re-test HBaseClusterTest on a live cluster. REVISION DETAIL https://reviews.facebook.net/D549 AFFECTED FILES pom.xml src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java src/main/java/org/apache/hadoop/hbase/util/HMerge.java src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java src/test/java/org/apache/hadoop/hbase/util/ClusterTestUtils.java src/test/java/org/apache/hadoop/hbase/util/DataGenerator.java src/test/java/org/apache/hadoop/hbase/util/HBaseClusterTest.java src/test/java/org/apache/hadoop/hbase/util/HdfsAppender.java src/test/java/org/apache/hadoop/hbase/util/KillProcessesAndVerify.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java HBase cluster test tool (port from 0.89-fb) --- Key: HBASE-4908 URL: https://issues.apache.org/jira/browse/HBASE-4908 Project: HBase Issue Type: Sub-task Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D549.1.patch, D549.2.patch, D549.3.patch Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk. I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine). The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.: hbase org.apache.hadoop.hbase.manual.HBaseTest -load 10:50:100:20 -tn load_test -read 1:10:50:20 -zk zk_quorum -bloom ROWCOL -compression GZIP I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4863) Make Thrift server thread pool bounded and add a command-line UI test
[ https://issues.apache.org/jira/browse/HBASE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161526#comment-13161526 ] Phabricator commented on HBASE-4863: mbautin has abandoned the revision [jira] [HBASE-4863] Make HBase Thrift server more configurable and add a command-line UI test. This diff was committed by @TedYu (http://svn.apache.org/repos/asf/hbase/trunk@1206267, http://svn.apache.org/repos/asf/hbase/trunk@1206369). REVISION DETAIL https://reviews.facebook.net/D531 Make Thrift server thread pool bounded and add a command-line UI test - Key: HBASE-4863 URL: https://issues.apache.org/jira/browse/HBASE-4863 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Fix For: 0.94.0 Attachments: 0001-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch, 0002-Fix-thread-leaks-in-the-HBase-thread-pool-server.patch, 4863.addendum, D531.1.patch, D531.2.patch, D531.3.patch, D531.4.patch This started as an internal hotfix where we found out that the Thrift server spawned 15000 threads. To bound the thread pool size I added a custom thread pool server implementation called HBaseThreadPoolServer into HBase codebase, and made the following parameters configurable from both command line and as config settings: minWorkerThreads, maxWorkerThreads, and maxQueuedRequests. Under an increasing load, the server creates new threads for every connection before the pool size reaches minWorkerThreads. After that, the server puts new connections into the queue and only creates a new thread when the queue is full. If an attempt to create a new thread fails, the server drops connection. The default TThreadPoolServer would crash in that case, but it never happened because the thread pool was unbounded, so the server would hang indefinitely, consume a lot of memory, and cause huge latency spikes on the client side. Another part of this fix is refactoring and unit testing of the command-line part of the Thrift server. The logic there is sufficiently complicated, and the existing ThriftServer class does not test that part at all. The new TestThriftServerCmdLine test starts the Thrift server on a random port with various combinations of options and talks to it through the client API from another thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4934) Display Master server and Regionserver start time on respective info servers.
Display Master server and Regionserver start time on respective info servers. - Key: HBASE-4934 URL: https://issues.apache.org/jira/browse/HBASE-4934 Project: HBase Issue Type: Improvement Reporter: Jonathan Hsieh Priority: Minor With operations like rolling restart or master failovers, it is difficult to tell if a server is the old instance or the new restarted instance. Adding a start date stamp on the info web pages would be helpful for determining this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4930) Reformat HBase home page
[ https://issues.apache.org/jira/browse/HBASE-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-4930: - Attachment: faq.html The reason I'm pointing out the FAQ is because of the reorg I did on it yesterday. I'm attaching a generated version of it (though it's not on the site yet). I'm keeping the questions larger and I want to minimize clutter on it and treat it as a mini-index. 2 other categories I want to add to the FAQ are map reduce and getting involved. Question: should I mention NoSQL on the home page? Reformat HBase home page Key: HBASE-4930 URL: https://issues.apache.org/jira/browse/HBASE-4930 Project: HBase Issue Type: Improvement Components: documentation Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: faq.html, index.html, index_HBASE_4930.xml.patch Reformat the HBase home page. It's not a radical redesign but breaks the information that is there into clearer sections: * Welcome to HBase! (it's always good to be polite to the reader) * When Should I Use HBase? * Features * How Can I Get More Information? * News I also updated the feature-list in this page (added some things, clarified a few things, removed one or two things) Note: I just overhauled the FAQ in the book today as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4930) Reformat HBase home page
[ https://issues.apache.org/jira/browse/HBASE-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161636#comment-13161636 ] Hadoop QA commented on HBASE-4930: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505822/index_HBASE_4930.xml.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. -1 javadoc. The javadoc tool appears to have generated -160 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 71 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestRollingRestart org.apache.hadoop.hbase.util.TestRegionSplitter org.apache.hadoop.hbase.client.TestMultiParallel org.apache.hadoop.hbase.master.TestRestartCluster org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler org.apache.hadoop.hbase.client.TestInstantSchemaChange org.apache.hadoop.hbase.regionserver.wal.TestHLogBench org.apache.hadoop.hbase.rest.TestGzipFilter org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD org.apache.hadoop.hbase.regionserver.TestAtomicOperation org.apache.hadoop.hbase.rest.TestScannersWithFilters org.apache.hadoop.hbase.TestInfoServers org.apache.hadoop.hbase.regionserver.TestParallelPut org.apache.hadoop.hbase.coprocessor.TestClassLoading org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.regionserver.wal.TestLogRolling org.apache.hadoop.hbase.filter.TestColumnRangeFilter org.apache.hadoop.hbase.mapred.TestTableInputFormat org.apache.hadoop.hbase.client.TestHCM org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildHole org.apache.hadoop.hbase.coprocessor.TestMasterObserver org.apache.hadoop.hbase.rest.TestStatusResource org.apache.hadoop.hbase.TestRegionRebalancing org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort org.apache.hadoop.hbase.rest.TestVersionResource org.apache.hadoop.hbase.client.TestScannerTimeout org.apache.hadoop.hbase.client.TestFromClientSide org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol org.apache.hadoop.hbase.rest.TestRowResource org.apache.hadoop.hbase.rest.TestScannerResource org.apache.hadoop.hbase.ipc.TestDelayedRpc org.apache.hadoop.hbase.rest.client.TestRemoteAdmin org.apache.hadoop.hbase.util.TestFSUtils org.apache.hadoop.hbase.master.TestDistributedLogSplitting org.apache.hadoop.hbase.rest.TestTableResource org.apache.hadoop.hbase.regionserver.wal.TestWALReplay org.apache.hadoop.hbase.master.TestHMasterRPCException org.apache.hadoop.hbase.util.TestIdLock org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster org.apache.hadoop.hbase.rest.TestTransform org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint org.apache.hadoop.hbase.client.TestInstantSchemaChangeSplit org.apache.hadoop.hbase.regionserver.TestHRegion org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyControl org.apache.hadoop.hbase.client.TestMultipleTimestamps org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort org.apache.hadoop.hbase.catalog.TestMetaReaderEditor org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed org.apache.hadoop.hbase.client.TestMetaScanner org.apache.hadoop.hbase.io.hfile.TestHFileBlock org.apache.hadoop.hbase.client.TestTimestampsFilter
[jira] [Created] (HBASE-4935) hbase 0.92.0 doesn't work going against 0.20.205.0, its packaged hadoop
hbase 0.92.0 doesn't work going against 0.20.205.0, its packaged hadoop --- Key: HBASE-4935 URL: https://issues.apache.org/jira/browse/HBASE-4935 Project: HBase Issue Type: Bug Reporter: stack See this Mikhail thread up on the list: http://search-hadoop.com/m/WMUZR24EAJ1/%2522SequenceFileLogReader+uses+a+reflection+hack+resulting+in+runtime+failures%2522subj=Re+SequenceFileLogReader+uses+a+reflection+hack+resulting+in+runtime+failures Dig into it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4935) hbase 0.92.0 doesn't work going against 0.20.205.0, its packaged hadoop
[ https://issues.apache.org/jira/browse/HBASE-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4935: - Priority: Blocker (was: Major) Fix Version/s: 0.92.0 hbase 0.92.0 doesn't work going against 0.20.205.0, its packaged hadoop --- Key: HBASE-4935 URL: https://issues.apache.org/jira/browse/HBASE-4935 Project: HBase Issue Type: Bug Reporter: stack Priority: Blocker Fix For: 0.92.0 See this Mikhail thread up on the list: http://search-hadoop.com/m/WMUZR24EAJ1/%2522SequenceFileLogReader+uses+a+reflection+hack+resulting+in+runtime+failures%2522subj=Re+SequenceFileLogReader+uses+a+reflection+hack+resulting+in+runtime+failures Dig into it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4936) Cached HRegionInterface connections crash when getting UnknownHost exceptions
Cached HRegionInterface connections crash when getting UnknownHost exceptions - Key: HBASE-4936 URL: https://issues.apache.org/jira/browse/HBASE-4936 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.92.0 Reporter: Andrei Dragomir This isssue is unlikely to come up in a cluster test case. However, for development, the following thing happens: 1. Start the HBase cluster locally, on network A (DNS A, etc) 2. The region locations are cached using the hostname (mycomputer.company.com, 211.x.y.z - real ip) 3. Change network location (go home) 4. Start the HBase cluster locally. My hostname / ips are not different (mycomputer, 192.168.0.130 - new ip) If the region locations have been cached using the hostname, there is an UnknownHostException in CatalogTracker.getCachedConnection(ServerName sn), uncaught in the catch statements. The server will crash constantly. The error should be caught and not rethrown, so that the cached connection expires normally. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4936) Cached HRegionInterface connections crash when getting UnknownHost exceptions
[ https://issues.apache.org/jira/browse/HBASE-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrei Dragomir updated HBASE-4936: --- Status: Patch Available (was: Open) Catch UnknownHostException Cached HRegionInterface connections crash when getting UnknownHost exceptions - Key: HBASE-4936 URL: https://issues.apache.org/jira/browse/HBASE-4936 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.92.0 Reporter: Andrei Dragomir This isssue is unlikely to come up in a cluster test case. However, for development, the following thing happens: 1. Start the HBase cluster locally, on network A (DNS A, etc) 2. The region locations are cached using the hostname (mycomputer.company.com, 211.x.y.z - real ip) 3. Change network location (go home) 4. Start the HBase cluster locally. My hostname / ips are not different (mycomputer, 192.168.0.130 - new ip) If the region locations have been cached using the hostname, there is an UnknownHostException in CatalogTracker.getCachedConnection(ServerName sn), uncaught in the catch statements. The server will crash constantly. The error should be caught and not rethrown, so that the cached connection expires normally. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4936) Cached HRegionInterface connections crash when getting UnknownHost exceptions
[ https://issues.apache.org/jira/browse/HBASE-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrei Dragomir updated HBASE-4936: --- Attachment: HBASE-4936.patch Cached HRegionInterface connections crash when getting UnknownHost exceptions - Key: HBASE-4936 URL: https://issues.apache.org/jira/browse/HBASE-4936 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.92.0 Reporter: Andrei Dragomir Attachments: HBASE-4936.patch This isssue is unlikely to come up in a cluster test case. However, for development, the following thing happens: 1. Start the HBase cluster locally, on network A (DNS A, etc) 2. The region locations are cached using the hostname (mycomputer.company.com, 211.x.y.z - real ip) 3. Change network location (go home) 4. Start the HBase cluster locally. My hostname / ips are not different (mycomputer, 192.168.0.130 - new ip) If the region locations have been cached using the hostname, there is an UnknownHostException in CatalogTracker.getCachedConnection(ServerName sn), uncaught in the catch statements. The server will crash constantly. The error should be caught and not rethrown, so that the cached connection expires normally. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4936) Cached HRegionInterface connections crash when getting UnknownHost exceptions
[ https://issues.apache.org/jira/browse/HBASE-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161677#comment-13161677 ] Hadoop QA commented on HBASE-4936: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505896/HBASE-4936.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/424//console This message is automatically generated. Cached HRegionInterface connections crash when getting UnknownHost exceptions - Key: HBASE-4936 URL: https://issues.apache.org/jira/browse/HBASE-4936 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.92.0 Reporter: Andrei Dragomir Attachments: HBASE-4936.patch This isssue is unlikely to come up in a cluster test case. However, for development, the following thing happens: 1. Start the HBase cluster locally, on network A (DNS A, etc) 2. The region locations are cached using the hostname (mycomputer.company.com, 211.x.y.z - real ip) 3. Change network location (go home) 4. Start the HBase cluster locally. My hostname / ips are not different (mycomputer, 192.168.0.130 - new ip) If the region locations have been cached using the hostname, there is an UnknownHostException in CatalogTracker.getCachedConnection(ServerName sn), uncaught in the catch statements. The server will crash constantly. The error should be caught and not rethrown, so that the cached connection expires normally. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4937) Error in Quick Start Shell Exercises
Error in Quick Start Shell Exercises
Key: HBASE-4937
URL: https://issues.apache.org/jira/browse/HBASE-4937
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Ryan Berdeen
The shell exercises in the Quick Start
(http://hbase.apache.org/book/quickstart.html) starts
{code}
hbase(main):003:0 create 'test', 'cf'
0 row(s) in 1.2200 seconds
hbase(main):003:0 list 'table'
test
1 row(s) in 0.0550 seconds
{code}
It looks like the second command is wrong. Running it, the actual output is
{code}
hbase(main):001:0 create 'test', 'cf'
0 row(s) in 0.3630 seconds
hbase(main):002:0 list 'table'
TABLE
0 row(s) in 0.0100 seconds
{code}
The argument to list should be 'test', not 'table', and the output in the
example is missing the {{TABLE}} line.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4934) Display Master server and Regionserver start time on respective info servers.
[ https://issues.apache.org/jira/browse/HBASE-4934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161743#comment-13161743 ] Jean-Daniel Cryans commented on HBASE-4934: --- Isn't that what the start code is? It only applies to region servers tho. Display Master server and Regionserver start time on respective info servers. - Key: HBASE-4934 URL: https://issues.apache.org/jira/browse/HBASE-4934 Project: HBase Issue Type: Improvement Reporter: Jonathan Hsieh Priority: Minor With operations like rolling restart or master failovers, it is difficult to tell if a server is the old instance or the new restarted instance. Adding a start date stamp on the info web pages would be helpful for determining this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4921) HTable initialization looks for EMPTY_START_ROW
[
https://issues.apache.org/jira/browse/HBASE-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161754#comment-13161754
]
Jean-Daniel Cryans commented on HBASE-4921:
---
My understanding is that this is used to check the existence of the table,
whether it's disabled or not shouldn't really matter. This is not the only
place where the region offline exception is thrown all the way up, admin.split
does the same thing.
What exactly was the stack trace you saw Pritam?
HTable initialization looks for EMPTY_START_ROW
---
Key: HBASE-4921
URL: https://issues.apache.org/jira/browse/HBASE-4921
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Pritam Damania
The HTable initialization does something like this :
{code}this.connection.locateRegion(tableName,
HConstants.EMPTY_START_ROW);{code}
What is the rationale behind this ? What would happen if this region is in
flight ? I ran into a problem where I disabled the first region of the table
and now I can't create an HTable instance to this table.
Disabling the first region is like disabling the entire table from a client
perspective. I feel this is not the correct behavior.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4934) Display Master server and Regionserver start time on respective info servers.
[ https://issues.apache.org/jira/browse/HBASE-4934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161761#comment-13161761 ] Jonathan Hsieh commented on HBASE-4934: --- Sure it is, but as a user I think in human readable dates (did the triggered restart happen 2 minutes ago?) instead of milliseconds since 1970... :) Display Master server and Regionserver start time on respective info servers. - Key: HBASE-4934 URL: https://issues.apache.org/jira/browse/HBASE-4934 Project: HBase Issue Type: Improvement Reporter: Jonathan Hsieh Priority: Minor With operations like rolling restart or master failovers, it is difficult to tell if a server is the old instance or the new restarted instance. Adding a start date stamp on the info web pages would be helpful for determining this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4337) Update HBase directory structure layout to be aligned with Hadoop
[ https://issues.apache.org/jira/browse/HBASE-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated HBASE-4337: -- Attachment: HBASE-4337-6.patch port patch work with the latest 0.92 rc tag Update HBase directory structure layout to be aligned with Hadoop - Key: HBASE-4337 URL: https://issues.apache.org/jira/browse/HBASE-4337 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.92.0 Reporter: Eric Yang Assignee: Eric Yang Attachments: HBASE-4337-1.patch, HBASE-4337-2.patch, HBASE-4337-3.patch, HBASE-4337-4.patch, HBASE-4337-5.patch, HBASE-4337-6.patch, HBASE-4337.patch In HADOOP-6255, a proposal was made for common directory layout for Hadoop ecosystem. This jira is to track the necessary work for making HBase directory structure aligned with Hadoop for better integration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4337) Update HBase directory structure layout to be aligned with Hadoop
[ https://issues.apache.org/jira/browse/HBASE-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161779#comment-13161779 ] Hadoop QA commented on HBASE-4337: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505917/HBASE-4337-6.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/425//console This message is automatically generated. Update HBase directory structure layout to be aligned with Hadoop - Key: HBASE-4337 URL: https://issues.apache.org/jira/browse/HBASE-4337 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.92.0 Reporter: Eric Yang Assignee: Eric Yang Attachments: HBASE-4337-1.patch, HBASE-4337-2.patch, HBASE-4337-3.patch, HBASE-4337-4.patch, HBASE-4337-5.patch, HBASE-4337-6.patch, HBASE-4337.patch In HADOOP-6255, a proposal was made for common directory layout for Hadoop ecosystem. This jira is to track the necessary work for making HBase directory structure aligned with Hadoop for better integration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4795) Fix TestHFileBlock when running on a 32-bit JVM
[ https://issues.apache.org/jira/browse/HBASE-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161783#comment-13161783 ] Phabricator commented on HBASE-4795: mbautin has abandoned the revision [jira] [HBASE-4795] Fix TestHFileBlock when running on a 32-bit JVM. The diff was committed, abandoning. REVISION DETAIL https://reviews.facebook.net/D459 Fix TestHFileBlock when running on a 32-bit JVM --- Key: HBASE-4795 URL: https://issues.apache.org/jira/browse/HBASE-4795 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Fix For: 0.94.0 Attachments: D459.1.patch, D459.2.patch, Fix-TestHFileBlock-heap-size-test-on-a-32-bit-JVM.patch Our Hudson test server seems to run a 32-bit JVM. This patch fixes TestHFileBlock to work correctly for both 64-bit and 32-bit JVM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4927) CatalogJanior:SplitParentFirstComparator doesn't sort as expected, for the last region when the endkey is empty
[ https://issues.apache.org/jira/browse/HBASE-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-4927: --- Attachment: 0001-HBASE-4927-CatalogJanior-SplitParentFirstComparator-.patch CatalogJanior:SplitParentFirstComparator doesn't sort as expected, for the last region when the endkey is empty --- Key: HBASE-4927 URL: https://issues.apache.org/jira/browse/HBASE-4927 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0, 0.94.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: 0001-HBASE-4927-CatalogJanior-SplitParentFirstComparator-.patch When reviewing HBASE-4238 backporting, Jon found this issue. What happens if the split points are (empty end key is the last key, empty start key is the first key) Parent [A,) L daughter [A,B), R daughter [B,) When sorted, we gets to end key comparision which results in this incorrector order: [A,B), [A,), [B,) we wanted: [A,), [A,B), [B,) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4927) CatalogJanior:SplitParentFirstComparator doesn't sort as expected, for the last region when the endkey is empty
[ https://issues.apache.org/jira/browse/HBASE-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-4927: --- Fix Version/s: 0.94.0 Status: Patch Available (was: Open) CatalogJanior:SplitParentFirstComparator doesn't sort as expected, for the last region when the endkey is empty --- Key: HBASE-4927 URL: https://issues.apache.org/jira/browse/HBASE-4927 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0, 0.94.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.94.0 Attachments: 0001-HBASE-4927-CatalogJanior-SplitParentFirstComparator-.patch When reviewing HBASE-4238 backporting, Jon found this issue. What happens if the split points are (empty end key is the last key, empty start key is the first key) Parent [A,) L daughter [A,B), R daughter [B,) When sorted, we gets to end key comparision which results in this incorrector order: [A,B), [A,), [B,) we wanted: [A,), [A,B), [B,) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4927) CatalogJanior:SplitParentFirstComparator doesn't sort as expected, for the last region when the endkey is empty
[ https://issues.apache.org/jira/browse/HBASE-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161791#comment-13161791 ] Jimmy Xiang commented on HBASE-4927: We do consider endkey actually. CatalogJanior:SplitParentFirstComparator doesn't sort as expected, for the last region when the endkey is empty --- Key: HBASE-4927 URL: https://issues.apache.org/jira/browse/HBASE-4927 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0, 0.94.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.94.0 Attachments: 0001-HBASE-4927-CatalogJanior-SplitParentFirstComparator-.patch When reviewing HBASE-4238 backporting, Jon found this issue. What happens if the split points are (empty end key is the last key, empty start key is the first key) Parent [A,) L daughter [A,B), R daughter [B,) When sorted, we gets to end key comparision which results in this incorrector order: [A,B), [A,), [B,) we wanted: [A,), [A,B), [B,) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4927) CatalogJanior:SplitParentFirstComparator doesn't sort as expected, for the last region when the endkey is empty
[ https://issues.apache.org/jira/browse/HBASE-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161792#comment-13161792 ] jirapos...@reviews.apache.org commented on HBASE-4927: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2992/ --- Review request for hbase and Michael Stack. Summary --- Some HRegionInfo related comparator doesn't consider empty end key. That's why there are some hacks in HRegionInfo's containsRange and containsRow method to check empty key. However, due to this issue, SplitParentFirstComparator has a bug. This addresses bug HBASE-4927. https://issues.apache.org/jira/browse/HBASE-4927 Diffs - src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 2a24d0b src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java b356b22 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java e047e30 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java baca241 Diff: https://reviews.apache.org/r/2992/diff Testing --- TestHRegionInfo and TestCatalogJanitor are geen. Thanks, Jimmy CatalogJanior:SplitParentFirstComparator doesn't sort as expected, for the last region when the endkey is empty --- Key: HBASE-4927 URL: https://issues.apache.org/jira/browse/HBASE-4927 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0, 0.94.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.94.0 Attachments: 0001-HBASE-4927-CatalogJanior-SplitParentFirstComparator-.patch When reviewing HBASE-4238 backporting, Jon found this issue. What happens if the split points are (empty end key is the last key, empty start key is the first key) Parent [A,) L daughter [A,B), R daughter [B,) When sorted, we gets to end key comparision which results in this incorrector order: [A,B), [A,), [B,) we wanted: [A,), [A,B), [B,) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4927) CatalogJanior:SplitParentFirstComparator doesn't sort as expected, for the last region when the endkey is empty
[ https://issues.apache.org/jira/browse/HBASE-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161803#comment-13161803 ] Hadoop QA commented on HBASE-4927: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505920/0001-HBASE-4927-CatalogJanior-SplitParentFirstComparator-.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -160 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 71 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestRollingRestart org.apache.hadoop.hbase.master.TestRestartCluster org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler org.apache.hadoop.hbase.master.TestDistributedLogSplitting org.apache.hadoop.hbase.master.TestHMasterRPCException org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed org.apache.hadoop.hbase.master.TestMaster org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable org.apache.hadoop.hbase.TestDrainingServer org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion org.apache.hadoop.hbase.TestFullLogReconstruction org.apache.hadoop.hbase.avro.TestAvroServer org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles org.apache.hadoop.hbase.master.TestMasterFailover org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.zookeeper.TestZooKeeperACL org.apache.hadoop.hbase.master.TestMasterTransitions org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat org.apache.hadoop.hbase.mapreduce.TestTableMapReduce org.apache.hadoop.hbase.master.TestOpenedRegionHandler Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/426//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/426//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/426//console This message is automatically generated. CatalogJanior:SplitParentFirstComparator doesn't sort as expected, for the last region when the endkey is empty --- Key: HBASE-4927 URL: https://issues.apache.org/jira/browse/HBASE-4927 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0, 0.94.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.94.0 Attachments: 0001-HBASE-4927-CatalogJanior-SplitParentFirstComparator-.patch When reviewing HBASE-4238 backporting, Jon found this issue. What happens if the split points are (empty end key is the last key, empty start key is the first key) Parent [A,) L daughter [A,B), R daughter [B,) When sorted, we gets to end key comparision which results in this incorrector order: [A,B), [A,), [B,) we wanted: [A,), [A,B), [B,) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4921) HTable initialization looks for EMPTY_START_ROW
[
https://issues.apache.org/jira/browse/HBASE-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161804#comment-13161804
]
Pritam Damania commented on HBASE-4921:
---
@Jean-Daniel Cryans
Here is the stack trace :
Exception in thread main
org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
VerificationTest_DummyTable,,1322854092301.58a6099fa8fd53eb23c6329cf9fc46fe.
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:839)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:694)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:661)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:137)
HTable initialization looks for EMPTY_START_ROW
---
Key: HBASE-4921
URL: https://issues.apache.org/jira/browse/HBASE-4921
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Pritam Damania
The HTable initialization does something like this :
{code}this.connection.locateRegion(tableName,
HConstants.EMPTY_START_ROW);{code}
What is the rationale behind this ? What would happen if this region is in
flight ? I ran into a problem where I disabled the first region of the table
and now I can't create an HTable instance to this table.
Disabling the first region is like disabling the entire table from a client
perspective. I feel this is not the correct behavior.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4844) Coprocessor hooks for log rolling
[ https://issues.apache.org/jira/browse/HBASE-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161808#comment-13161808 ] Lars Hofhansl commented on HBASE-4844: -- @stack: The hooks here would be specific to log rolling. There are already hooks that are called per log edit. My thought was that without much tighter integration into the core, there would be nothing useful the coprocessor hooks could do with the log *files* (can't move them, delete them, change them, as the rest of the system could not be altered accordingly to deal with that). But maybe you're right and the pre hooks should be added as well. Yes, adding a WALObserver would require a restart of the regionserver(s), there is no dynamic loading (as far as I can see) for WALObserver, currently. How do folks backup WALs now? Have some watcher process that watches the .logs and .oldlogs directory? Coprocessor hooks for log rolling - Key: HBASE-4844 URL: https://issues.apache.org/jira/browse/HBASE-4844 Project: HBase Issue Type: New Feature Affects Versions: 0.94.0 Reporter: Lars Hofhansl Priority: Minor In order to eventually do point in time recovery we need a way to reliably back up the logs. Rather than adding some hard coded changes, we can provide coprocessor hooks and folks can implement their own policies. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4938) Create a HRegion.getScanner public method that allows reading from a specified readPoint
Create a HRegion.getScanner public method that allows reading from a specified readPoint Key: HBASE-4938 URL: https://issues.apache.org/jira/browse/HBASE-4938 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Priority: Minor There is an existing api HRegion.getScanner(Scan) that allows scanning a table. My proposal is to extend it to HRegion.getScanner(Scan, long readPoint) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4938) Create a HRegion.getScanner public method that allows reading from a specified readPoint
[ https://issues.apache.org/jira/browse/HBASE-4938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161838#comment-13161838 ] dhruba borthakur commented on HBASE-4938: - This is a very minor restarting of the code. We have some internal HRegion API that needs to scan based on a external readPoint. Of course, we could duplicate this piece of code from HRegion.java into our own code base, but in the spirit of making all our code changes compatible with Apache copen source, I would like to request the community to analyze this requested code change and approve of it. I will provide a patch. Create a HRegion.getScanner public method that allows reading from a specified readPoint Key: HBASE-4938 URL: https://issues.apache.org/jira/browse/HBASE-4938 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Priority: Minor There is an existing api HRegion.getScanner(Scan) that allows scanning a table. My proposal is to extend it to HRegion.getScanner(Scan, long readPoint) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4938) Create a HRegion.getScanner public method that allows reading from a specified readPoint
[ https://issues.apache.org/jira/browse/HBASE-4938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161840#comment-13161840 ] dhruba borthakur commented on HBASE-4938: - Code changes here: https://reviews.facebook.net/D591 Create a HRegion.getScanner public method that allows reading from a specified readPoint Key: HBASE-4938 URL: https://issues.apache.org/jira/browse/HBASE-4938 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Priority: Minor There is an existing api HRegion.getScanner(Scan) that allows scanning a table. My proposal is to extend it to HRegion.getScanner(Scan, long readPoint) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4930) Reformat HBase home page
[ https://issues.apache.org/jira/browse/HBASE-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161841#comment-13161841 ] Doug Meil commented on HBASE-4930: -- I didn't see too many squawks, so I'm going to go with this. Reformat HBase home page Key: HBASE-4930 URL: https://issues.apache.org/jira/browse/HBASE-4930 Project: HBase Issue Type: Improvement Components: documentation Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: faq.html, index.html, index_HBASE_4930.xml.patch Reformat the HBase home page. It's not a radical redesign but breaks the information that is there into clearer sections: * Welcome to HBase! (it's always good to be polite to the reader) * When Should I Use HBase? * Features * How Can I Get More Information? * News I also updated the feature-list in this page (added some things, clarified a few things, removed one or two things) Note: I just overhauled the FAQ in the book today as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4930) Reformat HBase home page
[ https://issues.apache.org/jira/browse/HBASE-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-4930: - Resolution: Fixed Status: Resolved (was: Patch Available) Reformat HBase home page Key: HBASE-4930 URL: https://issues.apache.org/jira/browse/HBASE-4930 Project: HBase Issue Type: Improvement Components: documentation Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: faq.html, index.html, index_HBASE_4930.xml.patch Reformat the HBase home page. It's not a radical redesign but breaks the information that is there into clearer sections: * Welcome to HBase! (it's always good to be polite to the reader) * When Should I Use HBase? * Features * How Can I Get More Information? * News I also updated the feature-list in this page (added some things, clarified a few things, removed one or two things) Note: I just overhauled the FAQ in the book today as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4939) [book] expanding architecture overview, added more to FAQ, and some reformatting in troubleshooting
[book] expanding architecture overview, added more to FAQ, and some reformatting in troubleshooting --- Key: HBASE-4939 URL: https://issues.apache.org/jira/browse/HBASE-4939 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor book.xml * Added an Overview section in Architecture * Moved one entry in the FAQ to the Architecture Overview * Added a few more links in FAQ troubleshooting.xml * Created new 'Resources' section in Troubleshooting * This points out the mailing lists, search-hadoop, and jira. * The mailing lists weren't mentioned in troubleshooting, and the last 2 were in 'external tools' which is not exactly what they are in the context of troubleshooting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4939) [book] expanding architecture overview, added more to FAQ, and some reformatting in troubleshooting
[ https://issues.apache.org/jira/browse/HBASE-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-4939: - Attachment: docbkx_HBASE_4939.patch [book] expanding architecture overview, added more to FAQ, and some reformatting in troubleshooting --- Key: HBASE-4939 URL: https://issues.apache.org/jira/browse/HBASE-4939 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_HBASE_4939.patch book.xml * Added an Overview section in Architecture * Moved one entry in the FAQ to the Architecture Overview * Added a few more links in FAQ troubleshooting.xml * Created new 'Resources' section in Troubleshooting * This points out the mailing lists, search-hadoop, and jira. * The mailing lists weren't mentioned in troubleshooting, and the last 2 were in 'external tools' which is not exactly what they are in the context of troubleshooting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4939) [book] expanding architecture overview, added more to FAQ, and some reformatting in troubleshooting
[ https://issues.apache.org/jira/browse/HBASE-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-4939: - Description: book.xml * Added an Overview section in Architecture (NoSQL? When should I use Hadoop? And the difference between HBase and HDFS) * Moved one entry in the FAQ to the Architecture Overview * Added a few more links in FAQ troubleshooting.xml * Created new 'Resources' section in Troubleshooting * This points out the mailing lists, search-hadoop, and jira. * The mailing lists weren't mentioned in troubleshooting, and the last 2 were in 'external tools' which is not exactly what they are in the context of troubleshooting. was: book.xml * Added an Overview section in Architecture * Moved one entry in the FAQ to the Architecture Overview * Added a few more links in FAQ troubleshooting.xml * Created new 'Resources' section in Troubleshooting * This points out the mailing lists, search-hadoop, and jira. * The mailing lists weren't mentioned in troubleshooting, and the last 2 were in 'external tools' which is not exactly what they are in the context of troubleshooting. [book] expanding architecture overview, added more to FAQ, and some reformatting in troubleshooting --- Key: HBASE-4939 URL: https://issues.apache.org/jira/browse/HBASE-4939 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_HBASE_4939.patch book.xml * Added an Overview section in Architecture (NoSQL? When should I use Hadoop? And the difference between HBase and HDFS) * Moved one entry in the FAQ to the Architecture Overview * Added a few more links in FAQ troubleshooting.xml * Created new 'Resources' section in Troubleshooting * This points out the mailing lists, search-hadoop, and jira. * The mailing lists weren't mentioned in troubleshooting, and the last 2 were in 'external tools' which is not exactly what they are in the context of troubleshooting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4939) [book] expanding architecture overview, added more to FAQ, and some reformatting in troubleshooting
[ https://issues.apache.org/jira/browse/HBASE-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-4939: - Status: Patch Available (was: Open) [book] expanding architecture overview, added more to FAQ, and some reformatting in troubleshooting --- Key: HBASE-4939 URL: https://issues.apache.org/jira/browse/HBASE-4939 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_HBASE_4939.patch book.xml * Added an Overview section in Architecture (NoSQL? When should I use Hadoop? And the difference between HBase and HDFS) * Moved one entry in the FAQ to the Architecture Overview * Added a few more links in FAQ troubleshooting.xml * Created new 'Resources' section in Troubleshooting * This points out the mailing lists, search-hadoop, and jira. * The mailing lists weren't mentioned in troubleshooting, and the last 2 were in 'external tools' which is not exactly what they are in the context of troubleshooting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4939) [book] expanding architecture overview, added more to FAQ, and some reformatting in troubleshooting
[ https://issues.apache.org/jira/browse/HBASE-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-4939: - Resolution: Fixed Status: Resolved (was: Patch Available) [book] expanding architecture overview, added more to FAQ, and some reformatting in troubleshooting --- Key: HBASE-4939 URL: https://issues.apache.org/jira/browse/HBASE-4939 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_HBASE_4939.patch book.xml * Added an Overview section in Architecture (NoSQL? When should I use Hadoop? And the difference between HBase and HDFS) * Moved one entry in the FAQ to the Architecture Overview * Added a few more links in FAQ troubleshooting.xml * Created new 'Resources' section in Troubleshooting * This points out the mailing lists, search-hadoop, and jira. * The mailing lists weren't mentioned in troubleshooting, and the last 2 were in 'external tools' which is not exactly what they are in the context of troubleshooting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4379) [hbck] Does not complain about tables with no end region [Z,]
[ https://issues.apache.org/jira/browse/HBASE-4379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-4379: -- Status: Open (was: Patch Available) [hbck] Does not complain about tables with no end region [Z,] - Key: HBASE-4379 URL: https://issues.apache.org/jira/browse/HBASE-4379 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.92.0, 0.90.5 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: 0001-HBASE-4379-hbck-does-not-complain-about-tables-with-.patch, hbase-4379.v2.patch hbck does not detect or have an error condition when the last region of a table is missing (end key != ''). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4379) [hbck] Does not complain about tables with no end region [Z,]
[ https://issues.apache.org/jira/browse/HBASE-4379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-4379: -- Status: Patch Available (was: Open) Attempting to get this patch to rerun (it's been idle for a month or so). [hbck] Does not complain about tables with no end region [Z,] - Key: HBASE-4379 URL: https://issues.apache.org/jira/browse/HBASE-4379 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.92.0, 0.90.5 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: 0001-HBASE-4379-hbck-does-not-complain-about-tables-with-.patch, hbase-4379.v2.patch hbck does not detect or have an error condition when the last region of a table is missing (end key != ''). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4847) Activate single jvm for small tests on jenkins
[ https://issues.apache.org/jira/browse/HBASE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161883#comment-13161883 ] nkeywal commented on HBASE-4847: All the stuff unable to create new native thread is clearly an issue in the env. The error in 30/Nov/11 19:58 is more interesting, but I don't reproduce it here, and I've run the code trough the debugger without finding a possible cause Activate single jvm for small tests on jenkins -- Key: HBASE-4847 URL: https://issues.apache.org/jira/browse/HBASE-4847 Project: HBase Issue Type: Improvement Components: build, test Affects Versions: 0.94.0 Environment: build Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4847_all.v4.patch, 4847_all.v5.patch, 4847_all.v6.patch, 4847_all.v6.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_pom.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v3.patch This will not revolutionate performances alone. We will win between 1 to 4 minutes. But we win as well: - it's a step for parallelizing the tests - new tests are less expensive as they do not create a new jvm: it's a continuous win - it will allow to push it on dev env while having the same env on dev on build, and 3 minutes are 10% of small + medium tests execution time. I will do a few submit patch to see if it works well before asking for the real commit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4847) Activate single jvm for small tests on jenkins
[ https://issues.apache.org/jira/browse/HBASE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4847: --- Status: Open (was: Patch Available) Activate single jvm for small tests on jenkins -- Key: HBASE-4847 URL: https://issues.apache.org/jira/browse/HBASE-4847 Project: HBase Issue Type: Improvement Components: build, test Affects Versions: 0.94.0 Environment: build Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4847_all.v4.patch, 4847_all.v5.patch, 4847_all.v6.patch, 4847_all.v6.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_pom.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v3.patch This will not revolutionate performances alone. We will win between 1 to 4 minutes. But we win as well: - it's a step for parallelizing the tests - new tests are less expensive as they do not create a new jvm: it's a continuous win - it will allow to push it on dev env while having the same env on dev on build, and 3 minutes are 10% of small + medium tests execution time. I will do a few submit patch to see if it works well before asking for the real commit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4847) Activate single jvm for small tests on jenkins
[ https://issues.apache.org/jira/browse/HBASE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4847: --- Status: Patch Available (was: Open) Activate single jvm for small tests on jenkins -- Key: HBASE-4847 URL: https://issues.apache.org/jira/browse/HBASE-4847 Project: HBase Issue Type: Improvement Components: build, test Affects Versions: 0.94.0 Environment: build Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4847_all.v4.patch, 4847_all.v5.patch, 4847_all.v6.patch, 4847_all.v6.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_pom.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v3.patch This will not revolutionate performances alone. We will win between 1 to 4 minutes. But we win as well: - it's a step for parallelizing the tests - new tests are less expensive as they do not create a new jvm: it's a continuous win - it will allow to push it on dev env while having the same env on dev on build, and 3 minutes are 10% of small + medium tests execution time. I will do a few submit patch to see if it works well before asking for the real commit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4847) Activate single jvm for small tests on jenkins
[ https://issues.apache.org/jira/browse/HBASE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4847: --- Attachment: 4847_all.v7.patch Activate single jvm for small tests on jenkins -- Key: HBASE-4847 URL: https://issues.apache.org/jira/browse/HBASE-4847 Project: HBase Issue Type: Improvement Components: build, test Affects Versions: 0.94.0 Environment: build Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4847_all.v4.patch, 4847_all.v5.patch, 4847_all.v6.patch, 4847_all.v6.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_pom.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v3.patch This will not revolutionate performances alone. We will win between 1 to 4 minutes. But we win as well: - it's a step for parallelizing the tests - new tests are less expensive as they do not create a new jvm: it's a continuous win - it will allow to push it on dev env while having the same env on dev on build, and 3 minutes are 10% of small + medium tests execution time. I will do a few submit patch to see if it works well before asking for the real commit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4847) Activate single jvm for small tests on jenkins
[ https://issues.apache.org/jira/browse/HBASE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161904#comment-13161904 ] Hadoop QA commented on HBASE-4847: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505938/4847_all.v7.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -160 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 71 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.TestDrainingServer Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/427//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/427//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/427//console This message is automatically generated. Activate single jvm for small tests on jenkins -- Key: HBASE-4847 URL: https://issues.apache.org/jira/browse/HBASE-4847 Project: HBase Issue Type: Improvement Components: build, test Affects Versions: 0.94.0 Environment: build Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4847_all.v4.patch, 4847_all.v5.patch, 4847_all.v6.patch, 4847_all.v6.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_pom.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v3.patch This will not revolutionate performances alone. We will win between 1 to 4 minutes. But we win as well: - it's a step for parallelizing the tests - new tests are less expensive as they do not create a new jvm: it's a continuous win - it will allow to push it on dev env while having the same env on dev on build, and 3 minutes are 10% of small + medium tests execution time. I will do a few submit patch to see if it works well before asking for the real commit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4930) Reformat HBase home page
[ https://issues.apache.org/jira/browse/HBASE-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161912#comment-13161912 ] Hudson commented on HBASE-4930: --- Integrated in HBase-TRUNK #2509 (See [https://builds.apache.org/job/HBase-TRUNK/2509/]) hbase-4930. home page (index.html) reformat. Reformat HBase home page Key: HBASE-4930 URL: https://issues.apache.org/jira/browse/HBASE-4930 Project: HBase Issue Type: Improvement Components: documentation Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: faq.html, index.html, index_HBASE_4930.xml.patch Reformat the HBase home page. It's not a radical redesign but breaks the information that is there into clearer sections: * Welcome to HBase! (it's always good to be polite to the reader) * When Should I Use HBase? * Features * How Can I Get More Information? * News I also updated the feature-list in this page (added some things, clarified a few things, removed one or two things) Note: I just overhauled the FAQ in the book today as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4939) [book] expanding architecture overview, added more to FAQ, and some reformatting in troubleshooting
[ https://issues.apache.org/jira/browse/HBASE-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161913#comment-13161913 ] Hudson commented on HBASE-4939: --- Integrated in HBase-TRUNK #2509 (See [https://builds.apache.org/job/HBase-TRUNK/2509/]) hbase-4939 book.xml (architecture/faq), troubleshooting.xml (created resources section) [book] expanding architecture overview, added more to FAQ, and some reformatting in troubleshooting --- Key: HBASE-4939 URL: https://issues.apache.org/jira/browse/HBASE-4939 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_HBASE_4939.patch book.xml * Added an Overview section in Architecture (NoSQL? When should I use Hadoop? And the difference between HBase and HDFS) * Moved one entry in the FAQ to the Architecture Overview * Added a few more links in FAQ troubleshooting.xml * Created new 'Resources' section in Troubleshooting * This points out the mailing lists, search-hadoop, and jira. * The mailing lists weren't mentioned in troubleshooting, and the last 2 were in 'external tools' which is not exactly what they are in the context of troubleshooting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4940) hadoop-metrics.properties can include configuration of the rest context for ganglia
hadoop-metrics.properties can include configuration of the rest context for
ganglia
-
Key: HBASE-4940
URL: https://issues.apache.org/jira/browse/HBASE-4940
Project: HBase
Issue Type: Improvement
Components: metrics
Affects Versions: 0.90.5
Environment: HBase-0.90.1
Reporter: Mubarak Seyed
Priority: Minor
Fix For: 0.90.5
It appears from hadoop-metrics.properties that configuration for rest context
is missing. It would be good if we add the rest context and commented out them,
if anyone is using rest-server and if they want to monitor using ganglia
context then they can uncomment the rest context and use them for rest-server
monitoring using ganglia.
{code}
# Configuration of the rest context for ganglia
#rest.class=org.apache.hadoop.metrics.ganglia.GangliaContext
#rest.period=10
#rest.servers=ganglia-metad-hostname:port
{code}
Working on the patch, will submit it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4908) HBase cluster test tool (port from 0.89-fb)
[
https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HBASE-4908:
---
Attachment: D549.4.patch
mbautin updated the revision [jira] [HBASE-4908] HBase cluster test tool (port
from 0.89-fb).
Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA
Addressed some more of Stack's comments. Got rid of the kill test -- it is a
large area by itself and is best left out of the scope of this diff. Refactored
the common part of RestartMetaTest and HBaseClusterTest into
IntegrationTestTool. I think eventually we need to be able to run integration
tests both from command line and through the JUnit framework. Also, we can
probably unify the mini-cluster framework with the process-based HBase cluster
stuff later, and have a subset of integration tests run in both modes.
I also added two unit tests: TestMiniClusterLoad{Sequential,Parallel}. The
former runs a write workload followed by a separate read workload, and the
latter one runs a read workload concurrently with the writes (which is
noticeably slower -- maybe that's something to investigate).
A thorough round of testing and cleanup is still necessary before this diff
can be finalized.
REVISION DETAIL
https://reviews.facebook.net/D549
AFFECTED FILES
pom.xml
src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
src/main/java/org/apache/hadoop/hbase/util/HMerge.java
src/main/java/org/apache/hadoop/hbase/util/Keying.java
src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
src/test/java/org/apache/hadoop/hbase/util/ClusterTestUtils.java
src/test/java/org/apache/hadoop/hbase/util/HBaseClusterTest.java
src/test/java/org/apache/hadoop/hbase/util/HdfsAppender.java
src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java
src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java
src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java
src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
HBase cluster test tool (port from 0.89-fb)
---
Key: HBASE-4908
URL: https://issues.apache.org/jira/browse/HBASE-4908
Project: HBase
Issue Type: Sub-task
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments: D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch
Porting one of our HBase cluster test tools (a single-process multi-threaded
load generator and verifier) from 0.89-fb to trunk.
I cleaned up the code a bit compared to what's in 0.89-fb, and discovered
that it has some features that I have not tried yet (some kind of a kill
test, and some way to run HBase as multiple processes on one machine).
The main utility of this piece of code for us has been the HBaseClusterTest
command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a
load test in our five-node dev cluster testing, e.g.:
hbase org.apache.hadoop.hbase.manual.HBaseTest -load 10:50:100:20 -tn
load_test -read 1:10:50:20 -zk zk_quorum -bloom ROWCOL -compression
GZIP
I will be using this code
[jira] [Commented] (HBASE-4908) HBase cluster test tool (port from 0.89-fb)
[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13162041#comment-13162041 ] Phabricator commented on HBASE-4908: mbautin has commented on the revision [jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb). Some replies inline. I will remove HdfsAppender test as well -- it seems unrelated to the main purpose of the load tester tool. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:17 Moved to org.apache.hadoop.hbase.util. src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:92 Done. src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:103 Made 20 a constant and removed separate logic for the (endKey - startKey) = 1000 case. src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:204 Mentioned this in the class comment. src/test/java/org/apache/hadoop/hbase/manual/RestartMetaTest.java:36 Added. src/test/java/org/apache/hadoop/hbase/manual/RestartMetaTest.java:152 There is not a lot of overlap of functionality, but perhaps some overlap in structure, e.g. command-line argument processing. I'd like to avoid doing a lot of refactoring until the next iteration. Ideally it should be possible to run every integration test both as a JUnit test and as a command-line tool. REVISION DETAIL https://reviews.facebook.net/D549 HBase cluster test tool (port from 0.89-fb) --- Key: HBASE-4908 URL: https://issues.apache.org/jira/browse/HBASE-4908 Project: HBase Issue Type: Sub-task Reporter: Mikhail Bautin Assignee: Mikhail Bautin Attachments: D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk. I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine). The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.: hbase org.apache.hadoop.hbase.manual.HBaseTest -load 10:50:100:20 -tn load_test -read 1:10:50:20 -zk zk_quorum -bloom ROWCOL -compression GZIP I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4899) Region would be assigned twice easily with continually killing server and moving region in testing environment
[
https://issues.apache.org/jira/browse/HBASE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13162059#comment-13162059
]
Hudson commented on HBASE-4899:
---
Integrated in HBase-0.92-security #29 (See
[https://builds.apache.org/job/HBase-0.92-security/29/])
HBASE-4899 Region would be assigned twice easily with continually killing
server and moving region in testing environment
stack :
Files :
* /hbase/branches/0.92/CHANGES.txt
*
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java
Region would be assigned twice easily with continually killing server and
moving region in testing environment
---
Key: HBASE-4899
URL: https://issues.apache.org/jira/browse/HBASE-4899
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.92.1
Attachments: hbase-4899.patch, hbase-4899v2.patch, hbase-4899v3.patch
Before assigning region in ServerShutdownHandler#process, it will check
whether region is in RIT,
however, this checking doesn't work as the excepted in the following case:
1.move region A from server B to server C
2.kill server B
3.start server B immediately
Let's see what happen in the code for the above case
{code}
for step1:
1.1 server B close the region A,
1.2 master setOffline for region
A,(AssignmentManager#setOffline:this.regions.remove(regionInfo))
1.3 server C start to open region A.(Not completed)
for step3:
master ServerShutdownHandler#process() for server B
{
..
splitlog()
...
ListRegionState regionsInTransition =
this.services.getAssignmentManager()
.processServerShutdown(this.serverName);
...
Skip regions that were in transition unless CLOSING or PENDING_CLOSE
...
assign region
}
{code}
In fact, when running
ServerShutdownHandler#process()#this.services.getAssignmentManager().processServerShutdown(this.serverName),
region A is in RIT (step1.3 not completed), but the return ListRegionState
regionsInTransition doesn't contain it, because region A has removed from
AssignmentManager.regions by AssignmentManager#setOffline in step 1.2
Therefore, region A will be assigned twice.
Actually, one server killed and started twice will also easily cause region
assigned twice.
Exclude the above reason, another probability :
when execute ServerShutdownHandler#process()#MetaReader.getServerUserRegions
,region is included which is in RIT now.
But after completing MetaReader.getServerUserRegions, the region has been
opened in other server and is not in RIT now.
In our testing environment where balancing,moving and killing are executed
periodly, assigning region twice often happens, and it is hateful because it
will affect other test cases.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4903) Return a result from RegionObserver.preIncrement()
[ https://issues.apache.org/jira/browse/HBASE-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13162060#comment-13162060 ] Hudson commented on HBASE-4903: --- Integrated in HBase-0.92-security #29 (See [https://builds.apache.org/job/HBase-0.92-security/29/]) HBASE-4903 addendum for security build larsh : Files : * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java Return a result from RegionObserver.preIncrement() -- Key: HBASE-4903 URL: https://issues.apache.org/jira/browse/HBASE-4903 Project: HBase Issue Type: Improvement Reporter: Daniel Gómez Ferro Fix For: 0.94.0, 0.92.1 Attachments: 4903-add.txt, HBASE-4903-0.92.patch, HBASE-4903.patch, HBASE-4903.patch The only way to return a result from RegionObserver.preIncrement() is to use Result.readFields() after serializing the correct result. This can be fixed either returning a Result object from that function or adding setters to Result. Another option is to modify the parameters and receive a ListKeyValue as preGet() does. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4847) Activate single jvm for small tests on jenkins
[ https://issues.apache.org/jira/browse/HBASE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4847: --- Status: Open (was: Patch Available) Activate single jvm for small tests on jenkins -- Key: HBASE-4847 URL: https://issues.apache.org/jira/browse/HBASE-4847 Project: HBase Issue Type: Improvement Components: build, test Affects Versions: 0.94.0 Environment: build Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4847_all.v4.patch, 4847_all.v5.patch, 4847_all.v6.patch, 4847_all.v6.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v8.patch, 4847_pom.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v3.patch This will not revolutionate performances alone. We will win between 1 to 4 minutes. But we win as well: - it's a step for parallelizing the tests - new tests are less expensive as they do not create a new jvm: it's a continuous win - it will allow to push it on dev env while having the same env on dev on build, and 3 minutes are 10% of small + medium tests execution time. I will do a few submit patch to see if it works well before asking for the real commit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4847) Activate single jvm for small tests on jenkins
[ https://issues.apache.org/jira/browse/HBASE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4847: --- Attachment: 4847_all.v8.patch Activate single jvm for small tests on jenkins -- Key: HBASE-4847 URL: https://issues.apache.org/jira/browse/HBASE-4847 Project: HBase Issue Type: Improvement Components: build, test Affects Versions: 0.94.0 Environment: build Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4847_all.v4.patch, 4847_all.v5.patch, 4847_all.v6.patch, 4847_all.v6.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v8.patch, 4847_pom.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v3.patch This will not revolutionate performances alone. We will win between 1 to 4 minutes. But we win as well: - it's a step for parallelizing the tests - new tests are less expensive as they do not create a new jvm: it's a continuous win - it will allow to push it on dev env while having the same env on dev on build, and 3 minutes are 10% of small + medium tests execution time. I will do a few submit patch to see if it works well before asking for the real commit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4847) Activate single jvm for small tests on jenkins
[ https://issues.apache.org/jira/browse/HBASE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4847: --- Status: Patch Available (was: Open) Activate single jvm for small tests on jenkins -- Key: HBASE-4847 URL: https://issues.apache.org/jira/browse/HBASE-4847 Project: HBase Issue Type: Improvement Components: build, test Affects Versions: 0.94.0 Environment: build Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4847_all.v4.patch, 4847_all.v5.patch, 4847_all.v6.patch, 4847_all.v6.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v8.patch, 4847_pom.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v3.patch This will not revolutionate performances alone. We will win between 1 to 4 minutes. But we win as well: - it's a step for parallelizing the tests - new tests are less expensive as they do not create a new jvm: it's a continuous win - it will allow to push it on dev env while having the same env on dev on build, and 3 minutes are 10% of small + medium tests execution time. I will do a few submit patch to see if it works well before asking for the real commit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4847) Activate single jvm for small tests on jenkins
[ https://issues.apache.org/jira/browse/HBASE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13162065#comment-13162065 ] Hadoop QA commented on HBASE-4847: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505969/4847_all.v8.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/428//console This message is automatically generated. Activate single jvm for small tests on jenkins -- Key: HBASE-4847 URL: https://issues.apache.org/jira/browse/HBASE-4847 Project: HBase Issue Type: Improvement Components: build, test Affects Versions: 0.94.0 Environment: build Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4847_all.v4.patch, 4847_all.v5.patch, 4847_all.v6.patch, 4847_all.v6.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v8.patch, 4847_pom.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v3.patch This will not revolutionate performances alone. We will win between 1 to 4 minutes. But we win as well: - it's a step for parallelizing the tests - new tests are less expensive as they do not create a new jvm: it's a continuous win - it will allow to push it on dev env while having the same env on dev on build, and 3 minutes are 10% of small + medium tests execution time. I will do a few submit patch to see if it works well before asking for the real commit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4847) Activate single jvm for small tests on jenkins
[ https://issues.apache.org/jira/browse/HBASE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4847: --- Attachment: 4847_all.v8.patch Activate single jvm for small tests on jenkins -- Key: HBASE-4847 URL: https://issues.apache.org/jira/browse/HBASE-4847 Project: HBase Issue Type: Improvement Components: build, test Affects Versions: 0.94.0 Environment: build Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4847_all.v4.patch, 4847_all.v5.patch, 4847_all.v6.patch, 4847_all.v6.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v8.patch, 4847_all.v8.patch, 4847_pom.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v3.patch This will not revolutionate performances alone. We will win between 1 to 4 minutes. But we win as well: - it's a step for parallelizing the tests - new tests are less expensive as they do not create a new jvm: it's a continuous win - it will allow to push it on dev env while having the same env on dev on build, and 3 minutes are 10% of small + medium tests execution time. I will do a few submit patch to see if it works well before asking for the real commit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4847) Activate single jvm for small tests on jenkins
[ https://issues.apache.org/jira/browse/HBASE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4847: --- Status: Open (was: Patch Available) Activate single jvm for small tests on jenkins -- Key: HBASE-4847 URL: https://issues.apache.org/jira/browse/HBASE-4847 Project: HBase Issue Type: Improvement Components: build, test Affects Versions: 0.94.0 Environment: build Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4847_all.v4.patch, 4847_all.v5.patch, 4847_all.v6.patch, 4847_all.v6.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v7.patch, 4847_all.v8.patch, 4847_all.v8.patch, 4847_pom.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v3.patch This will not revolutionate performances alone. We will win between 1 to 4 minutes. But we win as well: - it's a step for parallelizing the tests - new tests are less expensive as they do not create a new jvm: it's a continuous win - it will allow to push it on dev env while having the same env on dev on build, and 3 minutes are 10% of small + medium tests execution time. I will do a few submit patch to see if it works well before asking for the real commit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
