[jira] [Updated] (HBASE-13122) return codes of some filters not efficent
[ https://issues.apache.org/jira/browse/HBASE-13122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13122: --- Status: Patch Available (was: Open) return codes of some filters not efficent - Key: HBASE-13122 URL: https://issues.apache.org/jira/browse/HBASE-13122 Project: HBase Issue Type: Improvement Components: Filters Affects Versions: 0.98.10.1, 0.94.24, 1.0.1 Reporter: Shuaifeng Zhou Attachments: 13122-master.patch, 13122.patch ColumnRangeFilter: when minColumnInclusive is false, it means all the cells at the current rowcolumn not fit the condition, so it should skip to next column, return code should be NEXT_COL, not SKIP. FamilyFilter is the similar sitution. Currently, SKIP will not causing error, but not efficent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-13082: -- Status: Patch Available (was: Open) Coarsen StoreScanner locks to RegionScanner --- Key: HBASE-13082 URL: https://issues.apache.org/jira/browse/HBASE-13082 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Attachments: 13082-test.txt, 13082.txt, 13082.txt Continuing where HBASE-10015 left of. We can avoid locking (and memory fencing) inside StoreScanner by deferring to the lock already held by the RegionScanner. In tests this shows quite a scan improvement and reduced CPU (the fences make the cores wait for memory fetches). There are some drawbacks too: * All calls to RegionScanner need to be remain synchronized * Implementors of coprocessors need to be diligent in following the locking contract. For example Phoenix does not lock RegionScanner.nextRaw() and required in the documentation (not picking on Phoenix, this one is my fault as I told them it's OK) * possible starving of flushes and compaction with heavy read load. RegionScanner operations would keep getting the locks and the flushes/compactions would not be able finalize the set of files. I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339788#comment-14339788 ] Lars Hofhansl commented on HBASE-13082: --- We have three problems: # StoreScanner is locked too often # If StoreScanner.next(ListCell) does not find any Cells (for example if they do not match timerange or filter) it will exhaust the entire store while holding the lock, preventing flushes/compactions from finishing # Client can timeout even though the server is still working, because the server does not currently indicate that it is working but just not returning anything. This patch is for #1. We can fix #2 in many cases by just returning an empty result after some number of iterations - but we *only* do that if we not found any Cells for the current row, otherwise we need to finish the row, i.e. find the next row (which of course could then exhaust the region if we're unlucky). But note that the solution for #2 would *clash* with this patch. With this patch it is no longer the lock on StoreScanner that protects it from concurrent flushes, but the synchronized on RegionScannerImpl, and that we cannot easily let without actually returning something back to the client. #3 would only work with HBASE-11544 since we still need to be able to guarantee entire rows to the client, but if we break out of the loops because we did not find any Cell after some time we do not know whether we do a whole row or not. So in reality all these things look like need to be fixed together. Given that neither #2 nor #3 can be satisfactorily fixed without HBASE-11544, I propose doing a bit more testing on patch, and then committing this here. Then we fix #3 (which would incidentally also fix #2 after HBASE-11544 is in). Coarsen StoreScanner locks to RegionScanner --- Key: HBASE-13082 URL: https://issues.apache.org/jira/browse/HBASE-13082 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Attachments: 13082-test.txt, 13082.txt Continuing where HBASE-10015 left of. We can avoid locking (and memory fencing) inside StoreScanner by deferring to the lock already held by the RegionScanner. In tests this shows quite a scan improvement and reduced CPU (the fences make the cores wait for memory fetches). There are some drawbacks too: * All calls to RegionScanner need to be remain synchronized * Implementors of coprocessors need to be diligent in following the locking contract. For example Phoenix does not lock RegionScanner.nextRaw() and required in the documentation (not picking on Phoenix, this one is my fault as I told them it's OK) * possible starving of flushes and compaction with heavy read load. RegionScanner operations would keep getting the locks and the flushes/compactions would not be able finalize the set of files. I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339802#comment-14339802 ] Lars Hofhansl commented on HBASE-13082: --- In other tests I find the gain about 41ns per row and per core (you can see how the results drift more with more cores in use above). Above I measured 66ns. So scanning 10m rows, we'd save 0.4-0.6s, 100m rows that'd be 4-6s, and with 1bn rows it'd add up to 40-60s. (Note that in tests above I scanned about 11m rows/s when using all 8 cores/HTs on my box) Coarsen StoreScanner locks to RegionScanner --- Key: HBASE-13082 URL: https://issues.apache.org/jira/browse/HBASE-13082 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Attachments: 13082-test.txt, 13082.txt, 13082.txt Continuing where HBASE-10015 left of. We can avoid locking (and memory fencing) inside StoreScanner by deferring to the lock already held by the RegionScanner. In tests this shows quite a scan improvement and reduced CPU (the fences make the cores wait for memory fetches). There are some drawbacks too: * All calls to RegionScanner need to be remain synchronized * Implementors of coprocessors need to be diligent in following the locking contract. For example Phoenix does not lock RegionScanner.nextRaw() and required in the documentation (not picking on Phoenix, this one is my fault as I told them it's OK) * possible starving of flushes and compaction with heavy read load. RegionScanner operations would keep getting the locks and the flushes/compactions would not be able finalize the set of files. I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13097) Use same EventLoopGroup for different AsyncRpcClients if possible
[
https://issues.apache.org/jira/browse/HBASE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339803#comment-14339803
]
Hadoop QA commented on HBASE-13097:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701268/HBASE-13097_1.patch
against master branch at commit 458846ef7b0528cb7952c413694eaf55c5d94342.
ATTACHMENT ID: 12701268
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12990//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12990//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12990//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12990//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12990//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12990//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12990//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12990//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12990//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12990//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12990//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12990//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12990//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12990//console
This message is automatically generated.
Use same EventLoopGroup for different AsyncRpcClients if possible
-
Key: HBASE-13097
URL: https://issues.apache.org/jira/browse/HBASE-13097
Project: HBase
Issue Type: Bug
Components: IPC/RPC, test
Affects Versions: 2.0.0, 1.1.0
Reporter: zhangduo
Assignee: zhangduo
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13097.patch, HBASE-13097_1.patch
In some unit tests(such as TestAcidGuarantees) we create multiple Connection
instance. If we use AsyncRpcClient, then there will be multiple netty
Bootstrap and every Bootstrap has its own PooledByteBufAllocator.
I haven't read the code clearly but it uses some threadlocal technics and
jmap shows io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry is the
biggest things on Heap.
See
https://builds.apache.org/job/HBase-TRUNK/6168/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestAcidGuarantees-output.txt
{noformat}
2015-02-24 23:50:29,704 WARN [JvmPauseMonitor]
util.JvmPauseMonitor$Monitor(167): Detected pause in JVM or host machine (eg
GC): pause of approximately 20133ms
GC pool 'PS MarkSweep' had collection(s):
[jira] [Updated] (HBASE-13100) Shell command to retrieve table splits
[
https://issues.apache.org/jira/browse/HBASE-13100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Singhi updated HBASE-13100:
--
Attachment: HBASE-13100.patch
Shell command to retrieve table splits
--
Key: HBASE-13100
URL: https://issues.apache.org/jira/browse/HBASE-13100
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Sean Busbey
Assignee: Ashish Singhi
Priority: Minor
Labels: beginner
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13100.patch
Add a shell command that returns the splits for a table.
Doing this yourself is currently possible, but involves going outside of the
public api.
{code}
jruby-1.7.3 :012 create 'example_table', 'f1', SPLITS = [10, 20, 30,
40]
0 row(s) in 0.5500 seconds
= Hbase::Table - example_table
jruby-1.7.3 :013
get_table('example_table').table.get_all_region_locations.map do |location|
org.apache.hadoop.hbase.util.Bytes::toStringBinary(location.get_region_info.get_start_key)
end
0 row(s) in 0.0130 seconds
= [, 10, 20, 30, 40]
jruby-1.7.3 :014
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13100) Shell command to retrieve table splits
[
https://issues.apache.org/jira/browse/HBASE-13100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339812#comment-14339812
]
Sean Busbey commented on HBASE-13100:
-
Does round-tripping work with the empty split point still present?
Shell command to retrieve table splits
--
Key: HBASE-13100
URL: https://issues.apache.org/jira/browse/HBASE-13100
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Sean Busbey
Assignee: Ashish Singhi
Priority: Minor
Labels: beginner
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13100.patch
Add a shell command that returns the splits for a table.
Doing this yourself is currently possible, but involves going outside of the
public api.
{code}
jruby-1.7.3 :012 create 'example_table', 'f1', SPLITS = [10, 20, 30,
40]
0 row(s) in 0.5500 seconds
= Hbase::Table - example_table
jruby-1.7.3 :013
get_table('example_table').table.get_all_region_locations.map do |location|
org.apache.hadoop.hbase.util.Bytes::toStringBinary(location.get_region_info.get_start_key)
end
0 row(s) in 0.0130 seconds
= [, 10, 20, 30, 40]
jruby-1.7.3 :014
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13100) Shell command to retrieve table splits
[
https://issues.apache.org/jira/browse/HBASE-13100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339827#comment-14339827
]
Sean Busbey commented on HBASE-13100:
-
{quote}
bq. Does round-tripping work with the empty split point still present?
Sorry, I did not get what you mean.
{quote}
Can you take the array of splits that comes back, pass it to a new table
creation, and end up with the same split points?
Shell command to retrieve table splits
--
Key: HBASE-13100
URL: https://issues.apache.org/jira/browse/HBASE-13100
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Sean Busbey
Assignee: Ashish Singhi
Priority: Minor
Labels: beginner
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13100.patch
Add a shell command that returns the splits for a table.
Doing this yourself is currently possible, but involves going outside of the
public api.
{code}
jruby-1.7.3 :012 create 'example_table', 'f1', SPLITS = [10, 20, 30,
40]
0 row(s) in 0.5500 seconds
= Hbase::Table - example_table
jruby-1.7.3 :013
get_table('example_table').table.get_all_region_locations.map do |location|
org.apache.hadoop.hbase.util.Bytes::toStringBinary(location.get_region_info.get_start_key)
end
0 row(s) in 0.0130 seconds
= [, 10, 20, 30, 40]
jruby-1.7.3 :014
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13084) Add labels to VisibilityLabelsCache asynchronously causes TestShell flakey
[
https://issues.apache.org/jira/browse/HBASE-13084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339835#comment-14339835
]
zhangduo commented on HBASE-13084:
--
[~ram_krish] Have you tried the new patch? Does it work for you? Thanks~
Add labels to VisibilityLabelsCache asynchronously causes TestShell flakey
--
Key: HBASE-13084
URL: https://issues.apache.org/jira/browse/HBASE-13084
Project: HBase
Issue Type: Bug
Components: test
Reporter: zhangduo
Assignee: zhangduo
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13084-addendum.patch, HBASE-13084.patch,
HBASE-13084_1.patch, HBASE-13084_2.patch, HBASE-13084_2.patch,
HBASE-13084_2.patch, HBASE-13084_2.patch, HBASE-13084_2_disable_test.patch
As discussed in HBASE-12953, we found this error in PreCommit log
https://builds.apache.org/job/PreCommit-HBASE-Build/12918/artifact/hbase-shell/target/surefire-reports/org.apache.hadoop.hbase.client.TestShell-output.txt
{noformat}
1) Error:
test_The_get/put_methods_should_work_for_data_written_with_Visibility(Hbase::VisibilityLabelsAdminMethodsTest):
ArgumentError: org.apache.hadoop.hbase.DoNotRetryIOException:
org.apache.hadoop.hbase.security.visibility.InvalidLabelException: Label
'TEST_VISIBILITY' doesn't exists
at
org.apache.hadoop.hbase.security.visibility.VisibilityController.setAuths(VisibilityController.java:808)
at
org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService$1.setAuths(VisibilityLabelsProtos.java:6036)
at
org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService.callMethod(VisibilityLabelsProtos.java:6219)
at
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:6867)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1707)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1689)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31309)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2038)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:744)
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-shell/src/main/ruby/hbase/visibility_labels.rb:84:in
`set_auths'
./src/test/ruby/hbase/visibility_labels_admin_test.rb:77:in
`test_The_get/put_methods_should_work_for_data_written_with_Visibility'
org/jruby/RubyProc.java:270:in `call'
org/jruby/RubyKernel.java:2105:in `send'
org/jruby/RubyArray.java:1620:in `each'
org/jruby/RubyArray.java:1620:in `each'
2) Error:
test_The_set/clear_methods_should_work_with_authorizations(Hbase::VisibilityLabelsAdminMethodsTest):
ArgumentError: No authentication set for the given user jenkins
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-shell/src/main/ruby/hbase/visibility_labels.rb:97:in
`get_auths'
./src/test/ruby/hbase/visibility_labels_admin_test.rb:57:in
`test_The_set/clear_methods_should_work_with_authorizations'
org/jruby/RubyProc.java:270:in `call'
org/jruby/RubyKernel.java:2105:in `send'
org/jruby/RubyArray.java:1620:in `each'
org/jruby/RubyArray.java:1620:in `each'
{noformat}
This is the test code
{code:title=visibility_labels_admin_test.rb}
label = 'TEST_VISIBILITY'
user = org.apache.hadoop.hbase.security.User.getCurrent().getName();
visibility_admin.add_labels(label)
visibility_admin.set_auths(user, label)
{code}
It says 'label does not exists' when calling set_auths.
Then I add some ugly logs in DefaultVisibilityLabelServiceImpl and
VisibilityLabelsCache.
{code:title=DefaultVisibilityLabelServiceImpl.java}
public OperationStatus[] addLabels(Listbyte[] labels) throws IOException {
...
if (mutateLabelsRegion(puts, finalOpStatus)) {
updateZk(true);
}
for (byte[] label : labels) {
String labelStr = Bytes.toString(label);
LOG.info(labelStr + = +
this.labelsCache.getLabelOrdinal(labelStr));
}
...
}
{code}
{code:title=VisibilityLabelsCache.java}
public void refreshLabelsCache(byte[] data) throws IOException {
LOG.info(refresh, new Exception());
...
}
{code}
And I modified TestVisibilityLabelsWithCustomVisLabService to use
DefaultVisibilityLabelServiceImpl, then collected the logs of setupBeforeClass
{noformat}
2015-02-21
[jira] [Commented] (HBASE-13124) Option to collect latencies for individual requests
[
https://issues.apache.org/jira/browse/HBASE-13124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339844#comment-14339844
]
Hari Krishna Dara commented on HBASE-13124:
---
[~busbey] I was thinking somethinng much simpler and attached a patch.
Option to collect latencies for individual requests
---
Key: HBASE-13124
URL: https://issues.apache.org/jira/browse/HBASE-13124
Project: HBase
Issue Type: Improvement
Components: Performance, test
Affects Versions: 0.98.6
Reporter: Hari Krishna Dara
Attachments: HBASE-13124-1.patch
Currently, the only option in {{PerformanceEvaluation}} is to print latency
percentile summary at the end, and it is also not suitable if we are testing
for a long duration (as all the timings are collected in memory). There
should be an option to dump the latencies with a timestamp as CSV to a file
so that this allows a detailed analysis on the generated data, such as:
- Percentile calculation at the desired precision on the latencies
- Latencies over time
- Histograms
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13112) quota.rb, security.rb and visibility_labels.rb leak connection
[ https://issues.apache.org/jira/browse/HBASE-13112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339852#comment-14339852 ] Hudson commented on HBASE-13112: FAILURE: Integrated in HBase-1.1 #217 (See [https://builds.apache.org/job/HBase-1.1/217/]) HBASE-13112 quota.rb, security.rb and visibility_labels.rb leak connection (Ashish Singhi) (tedyu: rev 0ff1418b32fc490993e20c51ff85879d2406ea75) * hbase-shell/src/main/ruby/hbase/hbase.rb * hbase-shell/src/main/ruby/hbase/security.rb * hbase-shell/src/main/ruby/hbase/visibility_labels.rb quota.rb, security.rb and visibility_labels.rb leak connection -- Key: HBASE-13112 URL: https://issues.apache.org/jira/browse/HBASE-13112 Project: HBase Issue Type: Bug Components: shell Affects Versions: 1.0.0, 2.0.0, 1.1.0 Reporter: Ashish Singhi Assignee: Ashish Singhi Attachments: HBASE-13112-branch-1.0.patch, HBASE-13112.patch quota.rb, security.rb and visibility_labels.rb leaks connection as noted by [~ndimiduk] [here|https://issues.apache.org/jira/browse/HBASE-12833?focusedCommentId=14279465page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14279465] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12975) SplitTranaction, RegionMergeTransaction to should have InterfaceAudience of LimitedPrivate(Coproc,Phoenix)
[
https://issues.apache.org/jira/browse/HBASE-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339746#comment-14339746
]
Sean Busbey commented on HBASE-12975:
-
If we make the LimitedPrivate Unstable for 1.0.z and then Evolving for 1.1.z+,
then we'd be fine. Quote from when this came up on HBASE-12972:
{quote}
bq. What if we marked Region LimitedPrivate / Unstable in 1.0.z?
Then why would someone use it? Region is supposed to be the supportable version
of HRegion. Unstable provides only effectively the same guarantees as
inappropriate use of Private interfaces.
{quote}
The reason to use it is that we'd label it Evolving or Stable in the next minor
release. We could even add a note in the javadoc explaining that it's Unstable
just to flag that folks have to keep in mind that it wasn't around at the start
of the API cut off for 1.0. Like the compat guide says, just because we have
the option to break things at a particular version (in this case at patches
sine it's Unstable) doesn't mean we will.
{quote}
I'd hope not, if perhaps only for this case. Otherwise we cannot extract
ourselves from this unfortunate situation until 1.1.0.
{quote}
Why don't we just plan to get 1.1.0 out the door in time for Phoenix 1.0? How
aggressive is the timeline?
SplitTranaction, RegionMergeTransaction to should have InterfaceAudience of
LimitedPrivate(Coproc,Phoenix)
--
Key: HBASE-12975
URL: https://issues.apache.org/jira/browse/HBASE-12975
Project: HBase
Issue Type: Improvement
Reporter: Rajeshbabu Chintaguntla
Assignee: Rajeshbabu Chintaguntla
Fix For: 2.0.0, 1.0.1, 1.1.0
Attachments: HBASE-12975.patch
Making SplitTransaction, RegionMergeTransaction limited private is required
to support local indexing feature in Phoenix to ensure regions colocation.
We can ensure region split, regions merge in the coprocessors in few method
calls without touching internals like creating zk's, file layout changes or
assignments.
1) stepsBeforePONR, stepsAfterPONR we can ensure split.
2) meta entries can pass through coprocessors to atomically update with the
normal split/merge.
3) rollback on failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13112) quota.rb, security.rb and visibility_labels.rb leak connection
[
https://issues.apache.org/jira/browse/HBASE-13112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339771#comment-14339771
]
Hadoop QA commented on HBASE-13112:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701276/HBASE-13112-branch-1.0.patch
against branch-1.0 branch at commit 458846ef7b0528cb7952c413694eaf55c5d94342.
ATTACHMENT ID: 12701276
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12992//console
This message is automatically generated.
quota.rb, security.rb and visibility_labels.rb leak connection
--
Key: HBASE-13112
URL: https://issues.apache.org/jira/browse/HBASE-13112
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 1.0.0, 2.0.0, 1.1.0
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Attachments: HBASE-13112-branch-1.0.patch, HBASE-13112.patch
quota.rb, security.rb and visibility_labels.rb leaks connection as noted by
[~ndimiduk]
[here|https://issues.apache.org/jira/browse/HBASE-12833?focusedCommentId=14279465page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14279465]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13123) Minor bug in ROW bloom filter
[ https://issues.apache.org/jira/browse/HBASE-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-13123: --- Status: Patch Available (was: Open) Minor bug in ROW bloom filter - Key: HBASE-13123 URL: https://issues.apache.org/jira/browse/HBASE-13123 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0, 1.1.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Minor Attachments: HBASE-13163.patch While checking the code for Bloom filter found that while checking if a key passes the ROW bloom check we try to create a bloom key. The bloom key should be constructed only with the row part of the key. But try to form the bloom key including the meta data part of the key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13123) Minor bug in ROW bloom filter
[ https://issues.apache.org/jira/browse/HBASE-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-13123: --- Attachment: HBASE-13163.patch A simple one line patch. Minor bug in ROW bloom filter - Key: HBASE-13123 URL: https://issues.apache.org/jira/browse/HBASE-13123 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0, 1.1.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Minor Attachments: HBASE-13163.patch While checking the code for Bloom filter found that while checking if a key passes the ROW bloom check we try to create a bloom key. The bloom key should be constructed only with the row part of the key. But try to form the bloom key including the meta data part of the key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-13082: -- Attachment: 13082.txt Reattaching patch for testing. Coarsen StoreScanner locks to RegionScanner --- Key: HBASE-13082 URL: https://issues.apache.org/jira/browse/HBASE-13082 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Attachments: 13082-test.txt, 13082.txt, 13082.txt Continuing where HBASE-10015 left of. We can avoid locking (and memory fencing) inside StoreScanner by deferring to the lock already held by the RegionScanner. In tests this shows quite a scan improvement and reduced CPU (the fences make the cores wait for memory fetches). There are some drawbacks too: * All calls to RegionScanner need to be remain synchronized * Implementors of coprocessors need to be diligent in following the locking contract. For example Phoenix does not lock RegionScanner.nextRaw() and required in the documentation (not picking on Phoenix, this one is my fault as I told them it's OK) * possible starving of flushes and compaction with heavy read load. RegionScanner operations would keep getting the locks and the flushes/compactions would not be able finalize the set of files. I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13097) Use same EventLoopGroup for different AsyncRpcClients if possible
[
https://issues.apache.org/jira/browse/HBASE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339804#comment-14339804
]
zhangduo commented on HBASE-13097:
--
Yeah, all green.
Use same EventLoopGroup for different AsyncRpcClients if possible
-
Key: HBASE-13097
URL: https://issues.apache.org/jira/browse/HBASE-13097
Project: HBase
Issue Type: Bug
Components: IPC/RPC, test
Affects Versions: 2.0.0, 1.1.0
Reporter: zhangduo
Assignee: zhangduo
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13097.patch, HBASE-13097_1.patch
In some unit tests(such as TestAcidGuarantees) we create multiple Connection
instance. If we use AsyncRpcClient, then there will be multiple netty
Bootstrap and every Bootstrap has its own PooledByteBufAllocator.
I haven't read the code clearly but it uses some threadlocal technics and
jmap shows io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry is the
biggest things on Heap.
See
https://builds.apache.org/job/HBase-TRUNK/6168/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestAcidGuarantees-output.txt
{noformat}
2015-02-24 23:50:29,704 WARN [JvmPauseMonitor]
util.JvmPauseMonitor$Monitor(167): Detected pause in JVM or host machine (eg
GC): pause of approximately 20133ms
GC pool 'PS MarkSweep' had collection(s): count=15 time=55525ms
{noformat}
Update: We use a singleton PooledByteBufAllocator so the reason should be too
many threads. So we will work on reduce the connections and rpclients in unit
tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13100) Shell command to retrieve table splits
[
https://issues.apache.org/jira/browse/HBASE-13100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339810#comment-14339810
]
Ashish Singhi commented on HBASE-13100:
---
Manual test result.
{noformat}
hbase(main):003:0 create 't2', 'f1', SPLITS = ['10', '20', '30', '40']
0 row(s) in 1.3820 seconds
= Hbase::Table - t2
hbase(main):004:0 get_splits 't2'
Total number of splits = 5
= [, 10, 20, 30, 40]
{noformat}
Shell command to retrieve table splits
--
Key: HBASE-13100
URL: https://issues.apache.org/jira/browse/HBASE-13100
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Sean Busbey
Assignee: Ashish Singhi
Priority: Minor
Labels: beginner
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13100.patch
Add a shell command that returns the splits for a table.
Doing this yourself is currently possible, but involves going outside of the
public api.
{code}
jruby-1.7.3 :012 create 'example_table', 'f1', SPLITS = [10, 20, 30,
40]
0 row(s) in 0.5500 seconds
= Hbase::Table - example_table
jruby-1.7.3 :013
get_table('example_table').table.get_all_region_locations.map do |location|
org.apache.hadoop.hbase.util.Bytes::toStringBinary(location.get_region_info.get_start_key)
end
0 row(s) in 0.0130 seconds
= [, 10, 20, 30, 40]
jruby-1.7.3 :014
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13097) Use same EventLoopGroup for different AsyncRpcClients if possible
[
https://issues.apache.org/jira/browse/HBASE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangduo updated HBASE-13097:
-
Attachment: HBASE-13097_2.patch
Add unit test for global EventLoopGroup
Use same EventLoopGroup for different AsyncRpcClients if possible
-
Key: HBASE-13097
URL: https://issues.apache.org/jira/browse/HBASE-13097
Project: HBase
Issue Type: Bug
Components: IPC/RPC, test
Affects Versions: 2.0.0, 1.1.0
Reporter: zhangduo
Assignee: zhangduo
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13097.patch, HBASE-13097_1.patch,
HBASE-13097_2.patch
In some unit tests(such as TestAcidGuarantees) we create multiple Connection
instance. If we use AsyncRpcClient, then there will be multiple netty
Bootstrap and every Bootstrap has its own PooledByteBufAllocator.
I haven't read the code clearly but it uses some threadlocal technics and
jmap shows io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry is the
biggest things on Heap.
See
https://builds.apache.org/job/HBase-TRUNK/6168/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestAcidGuarantees-output.txt
{noformat}
2015-02-24 23:50:29,704 WARN [JvmPauseMonitor]
util.JvmPauseMonitor$Monitor(167): Detected pause in JVM or host machine (eg
GC): pause of approximately 20133ms
GC pool 'PS MarkSweep' had collection(s): count=15 time=55525ms
{noformat}
Update: We use a singleton PooledByteBufAllocator so the reason should be too
many threads. So we will work on reduce the connections and rpclients in unit
tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13058) Hbase shell command 'scan' for non existent table shows unnecessary info for one unrelated existent table.
[
https://issues.apache.org/jira/browse/HBASE-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339837#comment-14339837
]
Srikanth Srungarapu commented on HBASE-13058:
-
Error message is looking nasty.
[~a72877] Just a thought. We can parse table name directly from exception
instead of relying on the arguments. Very similar to what is being done in case
of TableExists exception.
{code}
if cause.kind_of?(org.apache.hadoop.hbase.TableExistsException) then
str = java.lang.String.new(#{cause})
strs = str.split(\n)
if strs.size 0 then
s = strs[0].split(' ');
if(s.size 1)
raise Table already exists: #{s[1]}!
end
raise Table already exists: #{strs[0]}!
end
end
{code}
Hbase shell command 'scan' for non existent table shows unnecessary info for
one unrelated existent table.
--
Key: HBASE-13058
URL: https://issues.apache.org/jira/browse/HBASE-13058
Project: HBase
Issue Type: Bug
Components: Client
Reporter: Abhishek Kumar
Assignee: Abhishek Kumar
Priority: Trivial
Fix For: 2.0.0, 1.1.0
Attachments: 0001-HBASE-13058-Error-messages-in-scan-table.patch,
0001-HBASE-13058-shell-unknown-table-message-update.patch
When scanning for a non existent table in hbase shell, error message in shell
sometimes(based on META table content) displays completely unrelated table
info , which seems to be unnecessary and inconsistent with other error
messages:
{noformat}
hbase(main):016:0 scan 'noTable'
ROW COLUMN+CELL
ERROR: Unknown table Table 'noTable' was not found, got: hbase:namespace.!
-
hbase(main):017:0 scan '01_noTable'
ROW COLUMN+CELL
ERROR: Unknown table 01_noTable!
--
{noformat}
Its happening when doing a META table scan (to locate input table ) and
scanner stops at row of another table (beyond which table can not exist) in
ConnectionManager.locateRegionInMeta:
{noformat}
private RegionLocations locateRegionInMeta(TableName tableName, byte[] row,
boolean useCache, boolean retry, int replicaId) throws
IOException {
.
// possible we got a region of a different table...
if (!regionInfo.getTable().equals(tableName)) {
throw new TableNotFoundException(
Table ' + tableName + ' was not found, got: +
regionInfo.getTable() + .);
}
...
...
{noformat}
Here, we can simply put a debug message(if required) and just throw the
TableNotFoundException(tableName) with only tableName instead of with
scanner positioned row.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13124) Option to collect latencies for individual requests
[
https://issues.apache.org/jira/browse/HBASE-13124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339836#comment-14339836
]
Sean Busbey commented on HBASE-13124:
-
oooh. nice. Could we dump these as Avro?
Option to collect latencies for individual requests
---
Key: HBASE-13124
URL: https://issues.apache.org/jira/browse/HBASE-13124
Project: HBase
Issue Type: Improvement
Components: Performance, test
Affects Versions: 0.98.6
Reporter: Hari Krishna Dara
Currently, the only option in {{PerformanceEvaluation}} is to print latency
percentile summary at the end, and it is also not suitable if we are testing
for a long duration (as all the timings are collected in memory). There
should be an option to dump the latencies with a timestamp as CSV to a file
so that this allows a detailed analysis on the generated data, such as:
- Percentile calculation at the desired precision on the latencies
- Latencies over time
- Histograms
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13124) Option to collect latencies for individual requests
[
https://issues.apache.org/jira/browse/HBASE-13124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339838#comment-14339838
]
Sean Busbey commented on HBASE-13124:
-
Or would htrace support + a local trace-to-file receiver be enough?
Option to collect latencies for individual requests
---
Key: HBASE-13124
URL: https://issues.apache.org/jira/browse/HBASE-13124
Project: HBase
Issue Type: Improvement
Components: Performance, test
Affects Versions: 0.98.6
Reporter: Hari Krishna Dara
Currently, the only option in {{PerformanceEvaluation}} is to print latency
percentile summary at the end, and it is also not suitable if we are testing
for a long duration (as all the timings are collected in memory). There
should be an option to dump the latencies with a timestamp as CSV to a file
so that this allows a detailed analysis on the generated data, such as:
- Percentile calculation at the desired precision on the latencies
- Latencies over time
- Histograms
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13124) Option to collect latencies for individual requests
[
https://issues.apache.org/jira/browse/HBASE-13124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Krishna Dara updated HBASE-13124:
--
Attachment: HBASE-13124-1.patch
Patch adds {{--responseFile}} to turn on writing this data out to the specified
file. By default this only write a summary line every few milliseconds, but
with {{--detailedResponse}} option, it writes every single response.
Option to collect latencies for individual requests
---
Key: HBASE-13124
URL: https://issues.apache.org/jira/browse/HBASE-13124
Project: HBase
Issue Type: Improvement
Components: Performance, test
Affects Versions: 0.98.6
Reporter: Hari Krishna Dara
Attachments: HBASE-13124-1.patch
Currently, the only option in {{PerformanceEvaluation}} is to print latency
percentile summary at the end, and it is also not suitable if we are testing
for a long duration (as all the timings are collected in memory). There
should be an option to dump the latencies with a timestamp as CSV to a file
so that this allows a detailed analysis on the generated data, such as:
- Percentile calculation at the desired precision on the latencies
- Latencies over time
- Histograms
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13111) truncate_preserve command is failing with undefined method error
[
https://issues.apache.org/jira/browse/HBASE-13111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339760#comment-14339760
]
Ashish Singhi commented on HBASE-13111:
---
Thanks Ted.
truncate_preserve command is failing with undefined method error
Key: HBASE-13111
URL: https://issues.apache.org/jira/browse/HBASE-13111
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 1.0.0, 2.0.0, 1.1.0
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.0.1, 1.1.0
Attachments: HBASE-13111.patch
{noformat}
hbase(main):001:0 truncate_preserve 't1'
Truncating 't1' table (it may take a while):
ERROR: undefined method `getTable' for nil:NilClass
Here is some help for this command:
Disables, drops and recreates the specified table while still maintaing the
previous region boundaries.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13120) Allow disabling hadoop classpath and native library lookup
[
https://issues.apache.org/jira/browse/HBASE-13120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339800#comment-14339800
]
Sean Busbey commented on HBASE-13120:
-
{quote}
+if [ $DISABLE_HADOOP_CLASSPATH_LOOKUP != true ] ; then
{quote}
Please name this HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP. Not that I think we're
likely to conflict on the name, but just in case.
Also, if you just check if the string is non-zero length it'll be easier to
integrate programmatically determining if we should disable.
{quote}
I think this is a valid use case where there is hadoop installed on the
node(s), but we are trying to run HBase from a self-contained tarball. We
should not try to find the installed hadoop in this case.
{quote}
This is a very good point. How much more work would it be to get this exported
when we're starting local distributed mode? At a minimum we should add a note
to the ref guide telling folks to set it.
Allow disabling hadoop classpath and native library lookup
--
Key: HBASE-13120
URL: https://issues.apache.org/jira/browse/HBASE-13120
Project: HBase
Issue Type: Improvement
Components: hbase
Affects Versions: 0.98.0
Reporter: Siddharth Wagle
Attachments: HBASE-13120.patch
- Current bin/hbase script sets the java.library.path to include hadoop
native libs based on what version of hadoop is installed on the box.
{code}
HADOOP_IN_PATH=$(PATH=${HADOOP_HOME:-${HADOOP_PREFIX}}/bin:$PATH which
hadoop 2/dev/null)
{code}
- This effectively means that a self-contained HBase running with a different
version of embedded hadoop jars will fail to work in case of version
incompatibilities.
- Example: AMBARI-5707, runs a embedded HBase on local FS side-by-side with
cluster Hadoop installation.
- It would be good to have a hbase-env variable to completely override native
lib path or a config to disable native lib path lookup, in which case user
has to provide it during start.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13100) Shell command to retrieve table splits
[
https://issues.apache.org/jira/browse/HBASE-13100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Singhi updated HBASE-13100:
--
Status: Patch Available (was: Open)
Please review
Shell command to retrieve table splits
--
Key: HBASE-13100
URL: https://issues.apache.org/jira/browse/HBASE-13100
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Sean Busbey
Assignee: Ashish Singhi
Priority: Minor
Labels: beginner
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13100.patch
Add a shell command that returns the splits for a table.
Doing this yourself is currently possible, but involves going outside of the
public api.
{code}
jruby-1.7.3 :012 create 'example_table', 'f1', SPLITS = [10, 20, 30,
40]
0 row(s) in 0.5500 seconds
= Hbase::Table - example_table
jruby-1.7.3 :013
get_table('example_table').table.get_all_region_locations.map do |location|
org.apache.hadoop.hbase.util.Bytes::toStringBinary(location.get_region_info.get_start_key)
end
0 row(s) in 0.0130 seconds
= [, 10, 20, 30, 40]
jruby-1.7.3 :014
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13100) Shell command to retrieve table splits
[
https://issues.apache.org/jira/browse/HBASE-13100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339816#comment-14339816
]
Ashish Singhi commented on HBASE-13100:
---
Sorry, I did not get what you mean.
Shell command to retrieve table splits
--
Key: HBASE-13100
URL: https://issues.apache.org/jira/browse/HBASE-13100
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Sean Busbey
Assignee: Ashish Singhi
Priority: Minor
Labels: beginner
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13100.patch
Add a shell command that returns the splits for a table.
Doing this yourself is currently possible, but involves going outside of the
public api.
{code}
jruby-1.7.3 :012 create 'example_table', 'f1', SPLITS = [10, 20, 30,
40]
0 row(s) in 0.5500 seconds
= Hbase::Table - example_table
jruby-1.7.3 :013
get_table('example_table').table.get_all_region_locations.map do |location|
org.apache.hadoop.hbase.util.Bytes::toStringBinary(location.get_region_info.get_start_key)
end
0 row(s) in 0.0130 seconds
= [, 10, 20, 30, 40]
jruby-1.7.3 :014
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13123) Minor bug in ROW bloom filter
[
https://issues.apache.org/jira/browse/HBASE-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339813#comment-14339813
]
Hadoop QA commented on HBASE-13123:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701277/HBASE-13163.patch
against master branch at commit 458846ef7b0528cb7952c413694eaf55c5d94342.
ATTACHMENT ID: 12701277
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12993//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12993//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12993//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12993//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12993//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12993//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12993//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12993//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12993//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12993//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12993//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12993//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12993//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12993//console
This message is automatically generated.
Minor bug in ROW bloom filter
-
Key: HBASE-13123
URL: https://issues.apache.org/jira/browse/HBASE-13123
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 2.0.0, 1.1.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
Attachments: HBASE-13163.patch
While checking the code for Bloom filter found that while checking if a key
passes the ROW bloom check we try to create a bloom key. The bloom key should
be constructed only with the row part of the key. But try to form the bloom
key including the meta data part of the key.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13100) Shell command to retrieve table splits
[
https://issues.apache.org/jira/browse/HBASE-13100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339826#comment-14339826
]
Sean Busbey commented on HBASE-13100:
-
please make sure that folks can get the splits from the table object as well.
e.g.
{code}
hbase(main):003:0 create 't2', 'f1', SPLITS = ['10', '20', '30', '40']
0 row(s) in 1.3820 seconds
= Hbase::Table - t2
hbase(main):004:0 table = get_table 't2'
= Hbase::Table - t2
hbase(main):004:0 table.get_splits
Total number of splits = 5
= [, 10, 20, 30, 40]
{code}
You can use HBase::Table.add_shell_command to do this quickly in the definition
of get_splits (see shell/commands/count for an example)
Shell command to retrieve table splits
--
Key: HBASE-13100
URL: https://issues.apache.org/jira/browse/HBASE-13100
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Sean Busbey
Assignee: Ashish Singhi
Priority: Minor
Labels: beginner
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13100.patch
Add a shell command that returns the splits for a table.
Doing this yourself is currently possible, but involves going outside of the
public api.
{code}
jruby-1.7.3 :012 create 'example_table', 'f1', SPLITS = [10, 20, 30,
40]
0 row(s) in 0.5500 seconds
= Hbase::Table - example_table
jruby-1.7.3 :013
get_table('example_table').table.get_all_region_locations.map do |location|
org.apache.hadoop.hbase.util.Bytes::toStringBinary(location.get_region_info.get_start_key)
end
0 row(s) in 0.0130 seconds
= [, 10, 20, 30, 40]
jruby-1.7.3 :014
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12972) Region, a supportable public/evolving subset of HRegion
[
https://issues.apache.org/jira/browse/HBASE-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339741#comment-14339741
]
Sean Busbey commented on HBASE-12972:
-
AFAICT, these are all LimitedPrivate / Evolving so breaking is fine by me now
that we're limiting to 1.1.0+
Region, a supportable public/evolving subset of HRegion
---
Key: HBASE-12972
URL: https://issues.apache.org/jira/browse/HBASE-12972
Project: HBase
Issue Type: New Feature
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-12972-0.98.patch, HBASE-12972-0.98.patch,
HBASE-12972-0.98.patch, HBASE-12972-0.98.patch
On HBASE-12566, [~lhofhansl] proposed:
{quote}
Maybe we can have a {{Region}} interface that is to {{HRegion}} is what
{{Store}} is to {{HStore}}. Store marked with {{@InterfaceAudience.Private}}
but used in some coprocessor hooks.
{quote}
By example, now coprocessors have to reach into HRegion in order to
participate in row and region locking protocols, this is one area where the
functionality is legitimate for coprocessors but not for users, so an
in-between interface make sense.
In addition we should promote {{Store}}'s interface audience to
LimitedPrivate(COPROC).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13122) return codes of some filters not efficent
[ https://issues.apache.org/jira/browse/HBASE-13122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuaifeng Zhou updated HBASE-13122: --- Attachment: 13122-master.patch patch for master branch attached return codes of some filters not efficent - Key: HBASE-13122 URL: https://issues.apache.org/jira/browse/HBASE-13122 Project: HBase Issue Type: Improvement Components: Filters Affects Versions: 0.94.24, 1.0.1, 0.98.10.1 Reporter: Shuaifeng Zhou Attachments: 13122-master.patch, 13122.patch ColumnRangeFilter: when minColumnInclusive is false, it means all the cells at the current rowcolumn not fit the condition, so it should skip to next column, return code should be NEXT_COL, not SKIP. FamilyFilter is the similar sitution. Currently, SKIP will not causing error, but not efficent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13112) quota.rb, security.rb and visibility_labels.rb leak connection
[ https://issues.apache.org/jira/browse/HBASE-13112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-13112: -- Attachment: HBASE-13112-branch-1.0.patch Patch for branch-1* In branch-1* there is no quota.rb as it will commit only to master branch as part of HBASE-11598 Thanks, Ted quota.rb, security.rb and visibility_labels.rb leak connection -- Key: HBASE-13112 URL: https://issues.apache.org/jira/browse/HBASE-13112 Project: HBase Issue Type: Bug Components: shell Affects Versions: 1.0.0, 2.0.0, 1.1.0 Reporter: Ashish Singhi Assignee: Ashish Singhi Attachments: HBASE-13112-branch-1.0.patch, HBASE-13112.patch quota.rb, security.rb and visibility_labels.rb leaks connection as noted by [~ndimiduk] [here|https://issues.apache.org/jira/browse/HBASE-12833?focusedCommentId=14279465page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14279465] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13112) quota.rb, security.rb and visibility_labels.rb leak connection
[ https://issues.apache.org/jira/browse/HBASE-13112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13112: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the patch, Ashish. quota.rb, security.rb and visibility_labels.rb leak connection -- Key: HBASE-13112 URL: https://issues.apache.org/jira/browse/HBASE-13112 Project: HBase Issue Type: Bug Components: shell Affects Versions: 1.0.0, 2.0.0, 1.1.0 Reporter: Ashish Singhi Assignee: Ashish Singhi Attachments: HBASE-13112-branch-1.0.patch, HBASE-13112.patch quota.rb, security.rb and visibility_labels.rb leaks connection as noted by [~ndimiduk] [here|https://issues.apache.org/jira/browse/HBASE-12833?focusedCommentId=14279465page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14279465] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13123) Minor bug in ROW bloom filter
ramkrishna.s.vasudevan created HBASE-13123: -- Summary: Minor bug in ROW bloom filter Key: HBASE-13123 URL: https://issues.apache.org/jira/browse/HBASE-13123 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0, 1.1.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Minor While checking the code for Bloom filter found that while checking if a key passes the ROW bloom check we try to create a bloom key. The bloom key should be constructed only with the row part of the key. But try to form the bloom key including the meta data part of the key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13124) Option to collect latencies for individual requests
Hari Krishna Dara created HBASE-13124:
-
Summary: Option to collect latencies for individual requests
Key: HBASE-13124
URL: https://issues.apache.org/jira/browse/HBASE-13124
Project: HBase
Issue Type: Improvement
Components: Performance, test
Affects Versions: 0.98.6
Reporter: Hari Krishna Dara
Currently, the only option in {{PerformanceEvaluation}} is to print latency
percentile summary at the end, and it is also not suitable if we are testing
for a long duration (as all the timings are collected in memory). There should
be an option to dump the latencies with a timestamp as CSV to a file so that
this allows a detailed analysis on the generated data, such as:
- Percentile calculation at the desired precision on the latencies
- Latencies over time
- Histograms
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13124) Option to collect latencies for individual requests
[
https://issues.apache.org/jira/browse/HBASE-13124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339843#comment-14339843
]
Hari Krishna Dara commented on HBASE-13124:
---
Oops... read the options as {{\--responseFile}} and {{\--detailedResponse}}.
Option to collect latencies for individual requests
---
Key: HBASE-13124
URL: https://issues.apache.org/jira/browse/HBASE-13124
Project: HBase
Issue Type: Improvement
Components: Performance, test
Affects Versions: 0.98.6
Reporter: Hari Krishna Dara
Attachments: HBASE-13124-1.patch
Currently, the only option in {{PerformanceEvaluation}} is to print latency
percentile summary at the end, and it is also not suitable if we are testing
for a long duration (as all the timings are collected in memory). There
should be an option to dump the latencies with a timestamp as CSV to a file
so that this allows a detailed analysis on the generated data, such as:
- Percentile calculation at the desired precision on the latencies
- Latencies over time
- Histograms
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11544) [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME
[ https://issues.apache.org/jira/browse/HBASE-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339271#comment-14339271 ] Jonathan Lawlor commented on HBASE-11544: - Line length violations are in the auto generated protobuf classes [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME -- Key: HBASE-11544 URL: https://issues.apache.org/jira/browse/HBASE-11544 Project: HBase Issue Type: Bug Reporter: stack Assignee: Jonathan Lawlor Priority: Critical Labels: beginner Attachments: HBASE-11544-v1.patch, HBASE-11544-v2.patch, HBASE-11544-v3.patch, HBASE-11544-v4.patch, HBASE-11544-v5.patch, HBASE-11544-v6.patch, HBASE-11544-v6.patch, HBASE-11544-v6.patch Running some tests, I set hbase.client.scanner.caching=1000. Dataset has large cells. I kept OOME'ing. Serverside, we should measure how much we've accumulated and return to the client whatever we've gathered once we pass out a certain size threshold rather than keep accumulating till we OOME. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13119) FileLink should implement equals
[ https://issues.apache.org/jira/browse/HBASE-13119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-13119: -- Attachment: hbase-13119_v1.patch Here is a simple patch. [~mbertozzi], [~jmhsieh] mind a quick review? FileLink should implement equals Key: HBASE-13119 URL: https://issues.apache.org/jira/browse/HBASE-13119 Project: HBase Issue Type: Bug Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 2.0.0, 1.1.0 Attachments: hbase-13119_v1.patch StoreFileInfo used to implement Comparable, but got changed in HBASE-12749. Now comparing StoreFileInfo's with links results in false, since FileLink/HFileLink does not implement equals(). We should do both equals and hashCode() it seems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13120) Allow disabling hadoop classpath and native library lookup
Siddharth Wagle created HBASE-13120:
---
Summary: Allow disabling hadoop classpath and native library lookup
Key: HBASE-13120
URL: https://issues.apache.org/jira/browse/HBASE-13120
Project: HBase
Issue Type: Improvement
Components: hbase
Affects Versions: 0.98.0
Reporter: Siddharth Wagle
- Current bin/hbase script sets the java.library.path to include hadoop native
libs based on what version of hadoop is installed on the box.
{code}
HADOOP_IN_PATH=$(PATH=${HADOOP_HOME:-${HADOOP_PREFIX}}/bin:$PATH which hadoop
2/dev/null)
{code}
- This effectively means that a self-contained HBase running with a different
version of embedded hadoop jars will fail to work in case of version
incompatibilities.
- Example: AMBARI-5707, runs a embedded HBase on local FS side-by-side with
cluster Hadoop installation.
- It would be good to have a hbase-env variable to completely override native
lib path or a config to disable native lib path lookup, in which case user has
to provide it during start.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12924) HRegionServer#MovedRegionsCleaner Chore does not start
[
https://issues.apache.org/jira/browse/HBASE-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sameet Agarwal updated HBASE-12924:
---
Attachment: first-task-v3.patch
Replaced Thread.Sleep with a waitFor
HRegionServer#MovedRegionsCleaner Chore does not start
--
Key: HBASE-12924
URL: https://issues.apache.org/jira/browse/HBASE-12924
Project: HBase
Issue Type: Bug
Reporter: Jonathan Lawlor
Priority: Minor
Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.11
Attachments: first-task-v2.patch, first-task-v3.patch,
first-task.patch
This issue is very similar to the one described in HBASE-11354. The method
createAndStart in MovedRegionsCleaner creates an instance of the chore but
never starts the underlying thread:
{code}
static MovedRegionsCleaner createAndStart(HRegionServer rs){
Stoppable stoppable = new Stoppable() {
private volatile boolean isStopped = false;
@Override public void stop(String why) { isStopped = true;}
@Override public boolean isStopped() {return isStopped;}
};
return new MovedRegionsCleaner(rs, stoppable);
}
{code}
Nobody has noticed this issue so far so the Chore may not be that important
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13111) truncate_preserve command is failing with undefined method error
[
https://issues.apache.org/jira/browse/HBASE-13111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Singhi updated HBASE-13111:
--
Fix Version/s: 1.1.0
1.0.1
2.0.0
Status: Patch Available (was: Open)
truncate_preserve command is failing with undefined method error
Key: HBASE-13111
URL: https://issues.apache.org/jira/browse/HBASE-13111
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 1.0.0, 2.0.0, 1.1.0
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.0.1, 1.1.0
Attachments: HBASE-13111.patch
{noformat}
hbase(main):001:0 truncate_preserve 't1'
Truncating 't1' table (it may take a while):
ERROR: undefined method `getTable' for nil:NilClass
Here is some help for this command:
Disables, drops and recreates the specified table while still maintaing the
previous region boundaries.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13111) truncate_preserve command is failing with undefined method error
[
https://issues.apache.org/jira/browse/HBASE-13111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Singhi updated HBASE-13111:
--
Attachment: HBASE-13111.patch
truncate_preserve command is failing with undefined method error
Key: HBASE-13111
URL: https://issues.apache.org/jira/browse/HBASE-13111
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 1.0.0, 2.0.0, 1.1.0
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Attachments: HBASE-13111.patch
{noformat}
hbase(main):001:0 truncate_preserve 't1'
Truncating 't1' table (it may take a while):
ERROR: undefined method `getTable' for nil:NilClass
Here is some help for this command:
Disables, drops and recreates the specified table while still maintaing the
previous region boundaries.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13071) Hbase Streaming Scan Feature
[
https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338439#comment-14338439
]
Hadoop QA commented on HBASE-13071:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701068/HBASE-13071-v1.patch
against master branch at commit 1c957b65b16a8706caee140c18b84ea48a0dc0aa.
ATTACHMENT ID: 12701068
{color:red}-1 @author{color}. The patch appears to contain 2 @author tags
which the Hadoop community has agreed to not allow in code contributions.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12978//console
This message is automatically generated.
Hbase Streaming Scan Feature
Key: HBASE-13071
URL: https://issues.apache.org/jira/browse/HBASE-13071
Project: HBase
Issue Type: New Feature
Affects Versions: 0.98.11
Reporter: Eshcar Hillel
Attachments: HBASE-13071-v1.patch, HBaseStreamingScanDesign.pdf,
HbaseStreamingScanEvaluation.pdf
A scan operation iterates over all rows of a table or a subrange of the
table. The synchronous nature in which the data is served at the client side
hinders the speed the application traverses the data: it increases the
overall processing time, and may cause a great variance in the times the
application waits for the next piece of data.
The scanner next() method at the client side invokes an RPC to the
regionserver and then stores the results in a cache. The application can
specify how many rows will be transmitted per RPC; by default this is set to
100 rows.
The cache can be considered as a producer-consumer queue, where the hbase
client pushes the data to the queue and the application consumes it.
Currently this queue is synchronous, i.e., blocking. More specifically, when
the application consumed all the data from the cache --- so the cache is
empty --- the hbase client retrieves additional data from the server and
re-fills the cache with new data. During this time the application is blocked.
Under the assumption that the application processing time can be balanced by
the time it takes to retrieve the data, an asynchronous approach can reduce
the time the application is waiting for data.
We attach a design document.
We also have a patch that is based on a private branch, and some evaluation
results of this code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13112) quota.rb, security.rb and visibility_labels.rb leaks connection
[
https://issues.apache.org/jira/browse/HBASE-13112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338628#comment-14338628
]
Hadoop QA commented on HBASE-13112:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701063/HBASE-13112.patch
against master branch at commit 1c957b65b16a8706caee140c18b84ea48a0dc0aa.
ATTACHMENT ID: 12701063
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hbase.master.TestCatalogJanitor.testLastParentCleanedEvenIfDaughterGoneFirst(TestCatalogJanitor.java:550)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12977//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12977//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12977//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12977//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12977//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12977//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12977//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12977//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12977//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12977//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12977//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12977//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12977//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12977//console
This message is automatically generated.
quota.rb, security.rb and visibility_labels.rb leaks connection
---
Key: HBASE-13112
URL: https://issues.apache.org/jira/browse/HBASE-13112
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 1.0.0, 2.0.0, 1.1.0
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Attachments: HBASE-13112.patch
quota.rb, security.rb and visibility_labels.rb leaks connection as noted by
[~ndimiduk]
[here|https://issues.apache.org/jira/browse/HBASE-12833?focusedCommentId=14279465page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14279465]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13111) truncate_preserve command is failing with undefined method error
[
https://issues.apache.org/jira/browse/HBASE-13111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338636#comment-14338636
]
Hudson commented on HBASE-13111:
FAILURE: Integrated in HBase-TRUNK #6174 (See
[https://builds.apache.org/job/HBase-TRUNK/6174/])
HBASE-13111 truncate_preserve command is failing with undefined method error
(Ashish Singhi) (tedyu: rev 49b4f3737eb0dd7c5d88d6dcbe8b5d4f167c6a2b)
* hbase-shell/src/main/ruby/hbase/admin.rb
truncate_preserve command is failing with undefined method error
Key: HBASE-13111
URL: https://issues.apache.org/jira/browse/HBASE-13111
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 1.0.0, 2.0.0, 1.1.0
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.0.1, 1.1.0
Attachments: HBASE-13111.patch
{noformat}
hbase(main):001:0 truncate_preserve 't1'
Truncating 't1' table (it may take a while):
ERROR: undefined method `getTable' for nil:NilClass
Here is some help for this command:
Disables, drops and recreates the specified table while still maintaing the
previous region boundaries.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11544) [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME
[ https://issues.apache.org/jira/browse/HBASE-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Lawlor updated HBASE-11544: Attachment: HBASE-11544-v6.patch New patch to fix the tests that were failing due to the new return type from InteralScanner [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME -- Key: HBASE-11544 URL: https://issues.apache.org/jira/browse/HBASE-11544 Project: HBase Issue Type: Bug Reporter: stack Assignee: Jonathan Lawlor Priority: Critical Labels: beginner Attachments: HBASE-11544-v1.patch, HBASE-11544-v2.patch, HBASE-11544-v3.patch, HBASE-11544-v4.patch, HBASE-11544-v5.patch, HBASE-11544-v6.patch Running some tests, I set hbase.client.scanner.caching=1000. Dataset has large cells. I kept OOME'ing. Serverside, we should measure how much we've accumulated and return to the client whatever we've gathered once we pass out a certain size threshold rather than keep accumulating till we OOME. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12924) HRegionServer#MovedRegionsCleaner Chore does not start
[
https://issues.apache.org/jira/browse/HBASE-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sameet Agarwal updated HBASE-12924:
---
Attachment: first-task.patch
The patch adds the MovedRegionsCleaner chore to the background chores. Also
added a unit test to test this.
HRegionServer#MovedRegionsCleaner Chore does not start
--
Key: HBASE-12924
URL: https://issues.apache.org/jira/browse/HBASE-12924
Project: HBase
Issue Type: Bug
Reporter: Jonathan Lawlor
Priority: Minor
Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.11
Attachments: first-task.patch
This issue is very similar to the one described in HBASE-11354. The method
createAndStart in MovedRegionsCleaner creates an instance of the chore but
never starts the underlying thread:
{code}
static MovedRegionsCleaner createAndStart(HRegionServer rs){
Stoppable stoppable = new Stoppable() {
private volatile boolean isStopped = false;
@Override public void stop(String why) { isStopped = true;}
@Override public boolean isStopped() {return isStopped;}
};
return new MovedRegionsCleaner(rs, stoppable);
}
{code}
Nobody has noticed this issue so far so the Chore may not be that important
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11544) [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME
[ https://issues.apache.org/jira/browse/HBASE-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Lawlor updated HBASE-11544: Status: Patch Available (was: Open) [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME -- Key: HBASE-11544 URL: https://issues.apache.org/jira/browse/HBASE-11544 Project: HBase Issue Type: Bug Reporter: stack Assignee: Jonathan Lawlor Priority: Critical Labels: beginner Attachments: HBASE-11544-v1.patch, HBASE-11544-v2.patch, HBASE-11544-v3.patch, HBASE-11544-v4.patch, HBASE-11544-v5.patch, HBASE-11544-v6.patch Running some tests, I set hbase.client.scanner.caching=1000. Dataset has large cells. I kept OOME'ing. Serverside, we should measure how much we've accumulated and return to the client whatever we've gathered once we pass out a certain size threshold rather than keep accumulating till we OOME. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11544) [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME
[ https://issues.apache.org/jira/browse/HBASE-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Lawlor updated HBASE-11544: Status: Open (was: Patch Available) [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME -- Key: HBASE-11544 URL: https://issues.apache.org/jira/browse/HBASE-11544 Project: HBase Issue Type: Bug Reporter: stack Assignee: Jonathan Lawlor Priority: Critical Labels: beginner Attachments: HBASE-11544-v1.patch, HBASE-11544-v2.patch, HBASE-11544-v3.patch, HBASE-11544-v4.patch, HBASE-11544-v5.patch Running some tests, I set hbase.client.scanner.caching=1000. Dataset has large cells. I kept OOME'ing. Serverside, we should measure how much we've accumulated and return to the client whatever we've gathered once we pass out a certain size threshold rather than keep accumulating till we OOME. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13014) Java Tool For Region Moving
[ https://issues.apache.org/jira/browse/HBASE-13014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan updated HBASE-13014: --- Attachment: (was: HBASE-13014-v2.patch) Java Tool For Region Moving Key: HBASE-13014 URL: https://issues.apache.org/jira/browse/HBASE-13014 Project: HBase Issue Type: Improvement Reporter: Abhishek Singh Chouhan Assignee: Abhishek Singh Chouhan Attachments: HBASE-13014.patch As per discussion on HBASE-12989 we should move the functionality of region_mover.rb into a Java tool and use region_mover.rb only only as a wrapper around it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13071) Hbase Streaming Scan Feature
[ https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338596#comment-14338596 ] Eshcar Hillel commented on HBASE-13071: --- I tried running the code from master, but encountered problems (even without the patch). Specifically, when running commands from hbase shell wasn't able to exit the shell properly, therefore created the patch based on 0.98. Hbase Streaming Scan Feature Key: HBASE-13071 URL: https://issues.apache.org/jira/browse/HBASE-13071 Project: HBase Issue Type: New Feature Affects Versions: 0.98.11 Reporter: Eshcar Hillel Attachments: HBASE-13071-v1.patch, HBASE-13071-v2.patch, HBaseStreamingScanDesign.pdf, HbaseStreamingScanEvaluation.pdf A scan operation iterates over all rows of a table or a subrange of the table. The synchronous nature in which the data is served at the client side hinders the speed the application traverses the data: it increases the overall processing time, and may cause a great variance in the times the application waits for the next piece of data. The scanner next() method at the client side invokes an RPC to the regionserver and then stores the results in a cache. The application can specify how many rows will be transmitted per RPC; by default this is set to 100 rows. The cache can be considered as a producer-consumer queue, where the hbase client pushes the data to the queue and the application consumes it. Currently this queue is synchronous, i.e., blocking. More specifically, when the application consumed all the data from the cache --- so the cache is empty --- the hbase client retrieves additional data from the server and re-fills the cache with new data. During this time the application is blocked. Under the assumption that the application processing time can be balanced by the time it takes to retrieve the data, an asynchronous approach can reduce the time the application is waiting for data. We attach a design document. We also have a patch that is based on a private branch, and some evaluation results of this code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13014) Java Tool For Region Moving
[ https://issues.apache.org/jira/browse/HBASE-13014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan updated HBASE-13014: --- Attachment: HBASE-13014-v2.patch V2 with comments incorporated. Java Tool For Region Moving Key: HBASE-13014 URL: https://issues.apache.org/jira/browse/HBASE-13014 Project: HBase Issue Type: Improvement Reporter: Abhishek Singh Chouhan Assignee: Abhishek Singh Chouhan Attachments: HBASE-13014-v2.patch, HBASE-13014.patch As per discussion on HBASE-12989 we should move the functionality of region_mover.rb into a Java tool and use region_mover.rb only only as a wrapper around it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13014) Java Tool For Region Moving
[ https://issues.apache.org/jira/browse/HBASE-13014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan updated HBASE-13014: --- Attachment: HBASE-13014-v2.patch Java Tool For Region Moving Key: HBASE-13014 URL: https://issues.apache.org/jira/browse/HBASE-13014 Project: HBase Issue Type: Improvement Reporter: Abhishek Singh Chouhan Assignee: Abhishek Singh Chouhan Attachments: HBASE-13014-v2.patch, HBASE-13014.patch As per discussion on HBASE-12989 we should move the functionality of region_mover.rb into a Java tool and use region_mover.rb only only as a wrapper around it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13071) Hbase Streaming Scan Feature
[
https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338479#comment-14338479
]
Hadoop QA commented on HBASE-13071:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701075/HBASE-13071-v2.patch
against master branch at commit 1c957b65b16a8706caee140c18b84ea48a0dc0aa.
ATTACHMENT ID: 12701075
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12979//console
This message is automatically generated.
Hbase Streaming Scan Feature
Key: HBASE-13071
URL: https://issues.apache.org/jira/browse/HBASE-13071
Project: HBase
Issue Type: New Feature
Affects Versions: 0.98.11
Reporter: Eshcar Hillel
Attachments: HBASE-13071-v1.patch, HBASE-13071-v2.patch,
HBaseStreamingScanDesign.pdf, HbaseStreamingScanEvaluation.pdf
A scan operation iterates over all rows of a table or a subrange of the
table. The synchronous nature in which the data is served at the client side
hinders the speed the application traverses the data: it increases the
overall processing time, and may cause a great variance in the times the
application waits for the next piece of data.
The scanner next() method at the client side invokes an RPC to the
regionserver and then stores the results in a cache. The application can
specify how many rows will be transmitted per RPC; by default this is set to
100 rows.
The cache can be considered as a producer-consumer queue, where the hbase
client pushes the data to the queue and the application consumes it.
Currently this queue is synchronous, i.e., blocking. More specifically, when
the application consumed all the data from the cache --- so the cache is
empty --- the hbase client retrieves additional data from the server and
re-fills the cache with new data. During this time the application is blocked.
Under the assumption that the application processing time can be balanced by
the time it takes to retrieve the data, an asynchronous approach can reduce
the time the application is waiting for data.
We attach a design document.
We also have a patch that is based on a private branch, and some evaluation
results of this code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13098) HBase Connection Control
[ https://issues.apache.org/jira/browse/HBASE-13098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338484#comment-14338484 ] Ashish Singhi commented on HBASE-13098: --- bq. Please review current state of RPC and fold into your design what is missing. We already bound inbound traffic. Point out what is wrong w/ our approach. Thanks. [~saint@gmail.com], thanks for your comment. There is nothing in our approach. I went through HBASE-11598 and found that it limits the control only on few api's not all (correct me If I am wrong here) and I did not find any other RPC connection controllers. So came up with this. May be I can relate this one with [~ndimiduk] idea on HBASE-13101. HBase Connection Control Key: HBASE-13098 URL: https://issues.apache.org/jira/browse/HBASE-13098 Project: HBase Issue Type: New Feature Components: security Reporter: Ashish Singhi Assignee: Ashish Singhi Attachments: HBASE-13098.patch, HBase Connection Control.pdf It is desirable to set the limit on the number of client connections permitted to the HBase server by controlling with certain system variables/parameters. Too many connections to the HBase server imply too many queries and MR jobs running on HBase. This can slow down the performance of the system and lead to denial of service. Hence such connections need to be controlled. Using too many connections may just cause thrashing rather than get more useful work done. This is kind off inspired from http://www.ebaytechblog.com/2014/08/21/quality-of-service-in-hadoop/#.VO2JXXyUe9y -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13112) quota.rb, security.rb and visibility_labels.rb leaks connection
[ https://issues.apache.org/jira/browse/HBASE-13112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338520#comment-14338520 ] Ted Yu commented on HBASE-13112: +1 quota.rb, security.rb and visibility_labels.rb leaks connection --- Key: HBASE-13112 URL: https://issues.apache.org/jira/browse/HBASE-13112 Project: HBase Issue Type: Bug Components: shell Affects Versions: 1.0.0, 2.0.0, 1.1.0 Reporter: Ashish Singhi Assignee: Ashish Singhi Attachments: HBASE-13112.patch quota.rb, security.rb and visibility_labels.rb leaks connection as noted by [~ndimiduk] [here|https://issues.apache.org/jira/browse/HBASE-12833?focusedCommentId=14279465page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14279465] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13097) Reduce Connection and RpcClient creations in unit tests
[
https://issues.apache.org/jira/browse/HBASE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338524#comment-14338524
]
Hadoop QA commented on HBASE-13097:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701061/HBASE-13097.patch
against master branch at commit 1c957b65b16a8706caee140c18b84ea48a0dc0aa.
ATTACHMENT ID: 12701061
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hbase.http.TestHttpServerLifecycle.testStartedServerIsAlive(TestHttpServerLifecycle.java:71)
at
org.apache.hadoop.mapred.TestLazyOutput.testLazyOutput(TestLazyOutput.java:195)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12976//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12976//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12976//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12976//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12976//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12976//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12976//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12976//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12976//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12976//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12976//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12976//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12976//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12976//console
This message is automatically generated.
Reduce Connection and RpcClient creations in unit tests
---
Key: HBASE-13097
URL: https://issues.apache.org/jira/browse/HBASE-13097
Project: HBase
Issue Type: Bug
Components: IPC/RPC, test
Affects Versions: 2.0.0, 1.1.0
Reporter: zhangduo
Assignee: zhangduo
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13097.patch
In some unit tests(such as TestAcidGuarantees) we create multiple Connection
instance. If we use AsyncRpcClient, then there will be multiple netty
Bootstrap and every Bootstrap has its own PooledByteBufAllocator.
I haven't read the code clearly but it uses some threadlocal technics and
jmap shows
[jira] [Commented] (HBASE-12972) Region, a supportable public/evolving subset of HRegion
[
https://issues.apache.org/jira/browse/HBASE-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338592#comment-14338592
]
Sean Busbey commented on HBASE-12972:
-
Looks like the current patch makes changes that won't allow it in 1.0.z
(non-binary compat changes to existing LimitedPrivate.Evolving, adding new
features to LimitedPrivate.Evolving).
Can we aim for 0.98.y, 1.1.z, 2.y.z, or would folks prefer feedback to make it
compat?
Region, a supportable public/evolving subset of HRegion
---
Key: HBASE-12972
URL: https://issues.apache.org/jira/browse/HBASE-12972
Project: HBase
Issue Type: New Feature
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.11
Attachments: HBASE-12972-0.98.patch, HBASE-12972-0.98.patch,
HBASE-12972-0.98.patch
On HBASE-12566, [~lhofhansl] proposed:
{quote}
Maybe we can have a {{Region}} interface that is to {{HRegion}} is what
{{Store}} is to {{HStore}}. Store marked with {{@InterfaceAudience.Private}}
but used in some coprocessor hooks.
{quote}
By example, now coprocessors have to reach into HRegion in order to
participate in row and region locking protocols, this is one area where the
functionality is legitimate for coprocessors but not for users, so an
in-between interface make sense.
In addition we should promote {{Store}}'s interface audience to
LimitedPrivate(COPROC).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13097) Reduce Connection and RpcClient creations in unit tests
[
https://issues.apache.org/jira/browse/HBASE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338476#comment-14338476
]
Jurriaan Mous commented on HBASE-13097:
---
Yes it is a valid solution and would make having multiple connections a lot
cheaper. I am ok with it.
I think it would be very rare that anyone wants to have different Connections
with separate eventLoopGroups with different characteristics. This way it is
still possible.
Reduce Connection and RpcClient creations in unit tests
---
Key: HBASE-13097
URL: https://issues.apache.org/jira/browse/HBASE-13097
Project: HBase
Issue Type: Bug
Components: IPC/RPC, test
Affects Versions: 2.0.0, 1.1.0
Reporter: zhangduo
Assignee: zhangduo
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13097.patch
In some unit tests(such as TestAcidGuarantees) we create multiple Connection
instance. If we use AsyncRpcClient, then there will be multiple netty
Bootstrap and every Bootstrap has its own PooledByteBufAllocator.
I haven't read the code clearly but it uses some threadlocal technics and
jmap shows io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry is the
biggest things on Heap.
See
https://builds.apache.org/job/HBase-TRUNK/6168/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestAcidGuarantees-output.txt
{noformat}
2015-02-24 23:50:29,704 WARN [JvmPauseMonitor]
util.JvmPauseMonitor$Monitor(167): Detected pause in JVM or host machine (eg
GC): pause of approximately 20133ms
GC pool 'PS MarkSweep' had collection(s): count=15 time=55525ms
{noformat}
Update: We use a singleton PooledByteBufAllocator so the reason should be too
many threads. So we will work on reduce the connections and rpclients in unit
tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13111) truncate_preserve command is failing with undefined method error
[
https://issues.apache.org/jira/browse/HBASE-13111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338495#comment-14338495
]
Ted Yu commented on HBASE-13111:
+1
truncate_preserve command is failing with undefined method error
Key: HBASE-13111
URL: https://issues.apache.org/jira/browse/HBASE-13111
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 1.0.0, 2.0.0, 1.1.0
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.0.1, 1.1.0
Attachments: HBASE-13111.patch
{noformat}
hbase(main):001:0 truncate_preserve 't1'
Truncating 't1' table (it may take a while):
ERROR: undefined method `getTable' for nil:NilClass
Here is some help for this command:
Disables, drops and recreates the specified table while still maintaing the
previous region boundaries.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13097) Reduce Connection and RpcClient creations in unit tests
[
https://issues.apache.org/jira/browse/HBASE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338490#comment-14338490
]
Ted Yu commented on HBASE-13097:
lgtm
Reduce Connection and RpcClient creations in unit tests
---
Key: HBASE-13097
URL: https://issues.apache.org/jira/browse/HBASE-13097
Project: HBase
Issue Type: Bug
Components: IPC/RPC, test
Affects Versions: 2.0.0, 1.1.0
Reporter: zhangduo
Assignee: zhangduo
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13097.patch
In some unit tests(such as TestAcidGuarantees) we create multiple Connection
instance. If we use AsyncRpcClient, then there will be multiple netty
Bootstrap and every Bootstrap has its own PooledByteBufAllocator.
I haven't read the code clearly but it uses some threadlocal technics and
jmap shows io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry is the
biggest things on Heap.
See
https://builds.apache.org/job/HBase-TRUNK/6168/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestAcidGuarantees-output.txt
{noformat}
2015-02-24 23:50:29,704 WARN [JvmPauseMonitor]
util.JvmPauseMonitor$Monitor(167): Detected pause in JVM or host machine (eg
GC): pause of approximately 20133ms
GC pool 'PS MarkSweep' had collection(s): count=15 time=55525ms
{noformat}
Update: We use a singleton PooledByteBufAllocator so the reason should be too
many threads. So we will work on reduce the connections and rpclients in unit
tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13111) truncate_preserve command is failing with undefined method error
[
https://issues.apache.org/jira/browse/HBASE-13111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-13111:
---
Hadoop Flags: Reviewed
truncate_preserve command is failing with undefined method error
Key: HBASE-13111
URL: https://issues.apache.org/jira/browse/HBASE-13111
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 1.0.0, 2.0.0, 1.1.0
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.0.1, 1.1.0
Attachments: HBASE-13111.patch
{noformat}
hbase(main):001:0 truncate_preserve 't1'
Truncating 't1' table (it may take a while):
ERROR: undefined method `getTable' for nil:NilClass
Here is some help for this command:
Disables, drops and recreates the specified table while still maintaing the
previous region boundaries.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13111) truncate_preserve command is failing with undefined method error
[
https://issues.apache.org/jira/browse/HBASE-13111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-13111:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
truncate_preserve command is failing with undefined method error
Key: HBASE-13111
URL: https://issues.apache.org/jira/browse/HBASE-13111
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 1.0.0, 2.0.0, 1.1.0
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.0.1, 1.1.0
Attachments: HBASE-13111.patch
{noformat}
hbase(main):001:0 truncate_preserve 't1'
Truncating 't1' table (it may take a while):
ERROR: undefined method `getTable' for nil:NilClass
Here is some help for this command:
Disables, drops and recreates the specified table while still maintaing the
previous region boundaries.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13113) BufferedMutatorImpl should return currentWriteBufferSize in getWriteBufferSize()
[ https://issues.apache.org/jira/browse/HBASE-13113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Solomon Duskis updated HBASE-13113: --- Affects Version/s: 1.0.1 2.0.0 BufferedMutatorImpl should return currentWriteBufferSize in getWriteBufferSize() Key: HBASE-13113 URL: https://issues.apache.org/jira/browse/HBASE-13113 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 1.0.1 Reporter: Solomon Duskis -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13113) BufferedMutatorImpl should return currentWriteBufferSize in getWriteBufferSize()
Solomon Duskis created HBASE-13113: -- Summary: BufferedMutatorImpl should return currentWriteBufferSize in getWriteBufferSize() Key: HBASE-13113 URL: https://issues.apache.org/jira/browse/HBASE-13113 Project: HBase Issue Type: Bug Reporter: Solomon Duskis -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13111) truncate_preserve command is failing with undefined method error
[
https://issues.apache.org/jira/browse/HBASE-13111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338518#comment-14338518
]
Ted Yu commented on HBASE-13111:
Ran TestShell with patch which passed.
Thanks for the patch, Ashish.
truncate_preserve command is failing with undefined method error
Key: HBASE-13111
URL: https://issues.apache.org/jira/browse/HBASE-13111
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 1.0.0, 2.0.0, 1.1.0
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.0.1, 1.1.0
Attachments: HBASE-13111.patch
{noformat}
hbase(main):001:0 truncate_preserve 't1'
Truncating 't1' table (it may take a while):
ERROR: undefined method `getTable' for nil:NilClass
Here is some help for this command:
Disables, drops and recreates the specified table while still maintaing the
previous region boundaries.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13111) truncate_preserve command is failing with undefined method error
[
https://issues.apache.org/jira/browse/HBASE-13111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338593#comment-14338593
]
Hudson commented on HBASE-13111:
FAILURE: Integrated in HBase-1.1 #216 (See
[https://builds.apache.org/job/HBase-1.1/216/])
HBASE-13111 truncate_preserve command is failing with undefined method error
(Ashish Singhi) (tedyu: rev ac7faf3ef70930c1db49e0669fffc66a423b7412)
* hbase-shell/src/main/ruby/hbase/admin.rb
truncate_preserve command is failing with undefined method error
Key: HBASE-13111
URL: https://issues.apache.org/jira/browse/HBASE-13111
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 1.0.0, 2.0.0, 1.1.0
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.0.1, 1.1.0
Attachments: HBASE-13111.patch
{noformat}
hbase(main):001:0 truncate_preserve 't1'
Truncating 't1' table (it may take a while):
ERROR: undefined method `getTable' for nil:NilClass
Here is some help for this command:
Disables, drops and recreates the specified table while still maintaing the
previous region boundaries.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13071) Hbase Streaming Scan Feature
[ https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338571#comment-14338571 ] Ted Yu commented on HBASE-13071: For ClientAsyncPrefetchScanner.java and ClientSimpleScanner.java: please add license header add annotation for audience Mind putting patch on reviewboard ? Thanks Hbase Streaming Scan Feature Key: HBASE-13071 URL: https://issues.apache.org/jira/browse/HBASE-13071 Project: HBase Issue Type: New Feature Affects Versions: 0.98.11 Reporter: Eshcar Hillel Attachments: HBASE-13071-v1.patch, HBASE-13071-v2.patch, HBaseStreamingScanDesign.pdf, HbaseStreamingScanEvaluation.pdf A scan operation iterates over all rows of a table or a subrange of the table. The synchronous nature in which the data is served at the client side hinders the speed the application traverses the data: it increases the overall processing time, and may cause a great variance in the times the application waits for the next piece of data. The scanner next() method at the client side invokes an RPC to the regionserver and then stores the results in a cache. The application can specify how many rows will be transmitted per RPC; by default this is set to 100 rows. The cache can be considered as a producer-consumer queue, where the hbase client pushes the data to the queue and the application consumes it. Currently this queue is synchronous, i.e., blocking. More specifically, when the application consumed all the data from the cache --- so the cache is empty --- the hbase client retrieves additional data from the server and re-fills the cache with new data. During this time the application is blocked. Under the assumption that the application processing time can be balanced by the time it takes to retrieve the data, an asynchronous approach can reduce the time the application is waiting for data. We attach a design document. We also have a patch that is based on a private branch, and some evaluation results of this code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13071) Hbase Streaming Scan Feature
[ https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338581#comment-14338581 ] Eshcar Hillel commented on HBASE-13071: --- I use the following commands to create the patches git format-patch 0.98 --minimal --stdout HBASE-13071-v1.patch git diff --no-prefix 0.98 HBASE-13071-v2.patch Any idea why these can't be applied? Hbase Streaming Scan Feature Key: HBASE-13071 URL: https://issues.apache.org/jira/browse/HBASE-13071 Project: HBase Issue Type: New Feature Affects Versions: 0.98.11 Reporter: Eshcar Hillel Attachments: HBASE-13071-v1.patch, HBASE-13071-v2.patch, HBaseStreamingScanDesign.pdf, HbaseStreamingScanEvaluation.pdf A scan operation iterates over all rows of a table or a subrange of the table. The synchronous nature in which the data is served at the client side hinders the speed the application traverses the data: it increases the overall processing time, and may cause a great variance in the times the application waits for the next piece of data. The scanner next() method at the client side invokes an RPC to the regionserver and then stores the results in a cache. The application can specify how many rows will be transmitted per RPC; by default this is set to 100 rows. The cache can be considered as a producer-consumer queue, where the hbase client pushes the data to the queue and the application consumes it. Currently this queue is synchronous, i.e., blocking. More specifically, when the application consumed all the data from the cache --- so the cache is empty --- the hbase client retrieves additional data from the server and re-fills the cache with new data. During this time the application is blocked. Under the assumption that the application processing time can be balanced by the time it takes to retrieve the data, an asynchronous approach can reduce the time the application is waiting for data. We attach a design document. We also have a patch that is based on a private branch, and some evaluation results of this code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13071) Hbase Streaming Scan Feature
[
https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338590#comment-14338590
]
Ted Yu commented on HBASE-13071:
Applying patch v2 resulted in the following rejects:
{code}
-rw-r--r-- 1 tyu staff 1803 Feb 26 07:48
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java.rej
-rw-r--r-- 1 tyu staff 730 Feb 26 07:48
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientSmallScanner.java.rej
-rw-r--r-- 1 tyu staff 689 Feb 26 07:48
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java.rej
-rw-r--r-- 1 tyu staff 696 Feb 26 07:48
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/ReversedClientScanner.java.rej
-rw-r--r-- 1 tyu staff 1374 Feb 26 07:48
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java.rej
-rw-r--r-- 1 tyu staff 1504 Feb 26 07:48
./hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableConfiguration.java.rej
{code}
Please update your workspace to latest master.
Hbase Streaming Scan Feature
Key: HBASE-13071
URL: https://issues.apache.org/jira/browse/HBASE-13071
Project: HBase
Issue Type: New Feature
Affects Versions: 0.98.11
Reporter: Eshcar Hillel
Attachments: HBASE-13071-v1.patch, HBASE-13071-v2.patch,
HBaseStreamingScanDesign.pdf, HbaseStreamingScanEvaluation.pdf
A scan operation iterates over all rows of a table or a subrange of the
table. The synchronous nature in which the data is served at the client side
hinders the speed the application traverses the data: it increases the
overall processing time, and may cause a great variance in the times the
application waits for the next piece of data.
The scanner next() method at the client side invokes an RPC to the
regionserver and then stores the results in a cache. The application can
specify how many rows will be transmitted per RPC; by default this is set to
100 rows.
The cache can be considered as a producer-consumer queue, where the hbase
client pushes the data to the queue and the application consumes it.
Currently this queue is synchronous, i.e., blocking. More specifically, when
the application consumed all the data from the cache --- so the cache is
empty --- the hbase client retrieves additional data from the server and
re-fills the cache with new data. During this time the application is blocked.
Under the assumption that the application processing time can be balanced by
the time it takes to retrieve the data, an asynchronous approach can reduce
the time the application is waiting for data.
We attach a design document.
We also have a patch that is based on a private branch, and some evaluation
results of this code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13071) Hbase Streaming Scan Feature
[ https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338599#comment-14338599 ] Eshcar Hillel commented on HBASE-13071: --- I tried running the code from master, but encountered problems (even without the patch). Specifically, when running commands from hbase shell wasn't able to exit the shell properly, therefore created the patch based on 0.98. Hbase Streaming Scan Feature Key: HBASE-13071 URL: https://issues.apache.org/jira/browse/HBASE-13071 Project: HBase Issue Type: New Feature Affects Versions: 0.98.11 Reporter: Eshcar Hillel Attachments: HBASE-13071-v1.patch, HBASE-13071-v2.patch, HBaseStreamingScanDesign.pdf, HbaseStreamingScanEvaluation.pdf A scan operation iterates over all rows of a table or a subrange of the table. The synchronous nature in which the data is served at the client side hinders the speed the application traverses the data: it increases the overall processing time, and may cause a great variance in the times the application waits for the next piece of data. The scanner next() method at the client side invokes an RPC to the regionserver and then stores the results in a cache. The application can specify how many rows will be transmitted per RPC; by default this is set to 100 rows. The cache can be considered as a producer-consumer queue, where the hbase client pushes the data to the queue and the application consumes it. Currently this queue is synchronous, i.e., blocking. More specifically, when the application consumed all the data from the cache --- so the cache is empty --- the hbase client retrieves additional data from the server and re-fills the cache with new data. During this time the application is blocked. Under the assumption that the application processing time can be balanced by the time it takes to retrieve the data, an asynchronous approach can reduce the time the application is waiting for data. We attach a design document. We also have a patch that is based on a private branch, and some evaluation results of this code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13098) HBase Connection Control
[ https://issues.apache.org/jira/browse/HBASE-13098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338488#comment-14338488 ] Ashish Singhi commented on HBASE-13098: --- bq. There is nothing in our approach I meant there is nothing wrong in our approach. Sorry for the typo. HBase Connection Control Key: HBASE-13098 URL: https://issues.apache.org/jira/browse/HBASE-13098 Project: HBase Issue Type: New Feature Components: security Reporter: Ashish Singhi Assignee: Ashish Singhi Attachments: HBASE-13098.patch, HBase Connection Control.pdf It is desirable to set the limit on the number of client connections permitted to the HBase server by controlling with certain system variables/parameters. Too many connections to the HBase server imply too many queries and MR jobs running on HBase. This can slow down the performance of the system and lead to denial of service. Hence such connections need to be controlled. Using too many connections may just cause thrashing rather than get more useful work done. This is kind off inspired from http://www.ebaytechblog.com/2014/08/21/quality-of-service-in-hadoop/#.VO2JXXyUe9y -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13113) BufferedMutatorImpl should return currentWriteBufferSize in getWriteBufferSize()
[
https://issues.apache.org/jira/browse/HBASE-13113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338561#comment-14338561
]
Solomon Duskis commented on HBASE-13113:
This method in BufferedMutatorImpl is incorrect:
{code}
@Override
public long getWriteBufferSize() {
return this.writeBufferSize;
}
{code}
BufferedMutatorImpl should return currentWriteBufferSize in
getWriteBufferSize()
Key: HBASE-13113
URL: https://issues.apache.org/jira/browse/HBASE-13113
Project: HBase
Issue Type: Bug
Affects Versions: 2.0.0, 1.0.1
Reporter: Solomon Duskis
Assignee: Solomon Duskis
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-13113) BufferedMutatorImpl should return currentWriteBufferSize in getWriteBufferSize()
[ https://issues.apache.org/jira/browse/HBASE-13113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Solomon Duskis reassigned HBASE-13113: -- Assignee: Solomon Duskis BufferedMutatorImpl should return currentWriteBufferSize in getWriteBufferSize() Key: HBASE-13113 URL: https://issues.apache.org/jira/browse/HBASE-13113 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 1.0.1 Reporter: Solomon Duskis Assignee: Solomon Duskis -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13111) truncate_preserve command is failing with undefined method error
[
https://issues.apache.org/jira/browse/HBASE-13111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338522#comment-14338522
]
Hadoop QA commented on HBASE-13111:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701049/HBASE-13111.patch
against master branch at commit 1c957b65b16a8706caee140c18b84ea48a0dc0aa.
ATTACHMENT ID: 12701049
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12975//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12975//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12975//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12975//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12975//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12975//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12975//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12975//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12975//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12975//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12975//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12975//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12975//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12975//console
This message is automatically generated.
truncate_preserve command is failing with undefined method error
Key: HBASE-13111
URL: https://issues.apache.org/jira/browse/HBASE-13111
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 1.0.0, 2.0.0, 1.1.0
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.0.1, 1.1.0
Attachments: HBASE-13111.patch
{noformat}
hbase(main):001:0 truncate_preserve 't1'
Truncating 't1' table (it may take a while):
ERROR: undefined method `getTable' for nil:NilClass
Here is some help for this command:
Disables, drops and recreates the specified table while still maintaing the
previous region boundaries.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13112) quota.rb, security.rb and visibility_labels.rb leaks connection
Ashish Singhi created HBASE-13112: - Summary: quota.rb, security.rb and visibility_labels.rb leaks connection Key: HBASE-13112 URL: https://issues.apache.org/jira/browse/HBASE-13112 Project: HBase Issue Type: Bug Components: shell Affects Versions: 1.0.0, 2.0.0, 1.1.0 Reporter: Ashish Singhi Assignee: Ashish Singhi quota.rb, security.rb and visibility_labels.rb leaks connection as noted by [~ndimiduk] [here|https://issues.apache.org/jira/browse/HBASE-12833?focusedCommentId=14279465page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14279465] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13097) Reduce Connection and RpcClient creations in unit tests
[
https://issues.apache.org/jira/browse/HBASE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangduo updated HBASE-13097:
-
Fix Version/s: 1.1.0
2.0.0
Assignee: zhangduo
Status: Patch Available (was: Open)
Reduce Connection and RpcClient creations in unit tests
---
Key: HBASE-13097
URL: https://issues.apache.org/jira/browse/HBASE-13097
Project: HBase
Issue Type: Bug
Components: IPC/RPC, test
Affects Versions: 2.0.0, 1.1.0
Reporter: zhangduo
Assignee: zhangduo
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13097.patch
In some unit tests(such as TestAcidGuarantees) we create multiple Connection
instance. If we use AsyncRpcClient, then there will be multiple netty
Bootstrap and every Bootstrap has its own PooledByteBufAllocator.
I haven't read the code clearly but it uses some threadlocal technics and
jmap shows io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry is the
biggest things on Heap.
See
https://builds.apache.org/job/HBase-TRUNK/6168/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestAcidGuarantees-output.txt
{noformat}
2015-02-24 23:50:29,704 WARN [JvmPauseMonitor]
util.JvmPauseMonitor$Monitor(167): Detected pause in JVM or host machine (eg
GC): pause of approximately 20133ms
GC pool 'PS MarkSweep' had collection(s): count=15 time=55525ms
{noformat}
Update: We use a singleton PooledByteBufAllocator so the reason should be too
many threads. So we will work on reduce the connections and rpclients in unit
tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13071) Hbase Streaming Scan Feature
[ https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338438#comment-14338438 ] Eshcar Hillel commented on HBASE-13071: --- I've just attached the patch. The default value for scanners is sync. This can be easily changed. Addressing the issue raised by [~jonathan.lawlor]: The prefetch logic is the same for sync and async scanners. Therefore, async scanner stops RPCs if the max result size is exceeded. However, since the prefetch is executed in the background, it is possible that the size of the data inside the cache exceeds the max size set by the user (which cannot happen with sync scanner). There are ways to handle this, but this requires knowing the size of the data in the cache at any point and limiting the size of the data retrieved from the server with respect to this size. This may reduce the performance gain. I plan to attach a patch of the YCSB extension if anyone wants to re-run the experiments. Hbase Streaming Scan Feature Key: HBASE-13071 URL: https://issues.apache.org/jira/browse/HBASE-13071 Project: HBase Issue Type: New Feature Affects Versions: 0.98.11 Reporter: Eshcar Hillel Attachments: HBASE-13071-v1.patch, HBaseStreamingScanDesign.pdf, HbaseStreamingScanEvaluation.pdf A scan operation iterates over all rows of a table or a subrange of the table. The synchronous nature in which the data is served at the client side hinders the speed the application traverses the data: it increases the overall processing time, and may cause a great variance in the times the application waits for the next piece of data. The scanner next() method at the client side invokes an RPC to the regionserver and then stores the results in a cache. The application can specify how many rows will be transmitted per RPC; by default this is set to 100 rows. The cache can be considered as a producer-consumer queue, where the hbase client pushes the data to the queue and the application consumes it. Currently this queue is synchronous, i.e., blocking. More specifically, when the application consumed all the data from the cache --- so the cache is empty --- the hbase client retrieves additional data from the server and re-fills the cache with new data. During this time the application is blocked. Under the assumption that the application processing time can be balanced by the time it takes to retrieve the data, an asynchronous approach can reduce the time the application is waiting for data. We attach a design document. We also have a patch that is based on a private branch, and some evaluation results of this code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13071) Hbase Streaming Scan Feature
[ https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-13071: -- Affects Version/s: 0.98.11 Status: Patch Available (was: Open) Hbase Streaming Scan Feature Key: HBASE-13071 URL: https://issues.apache.org/jira/browse/HBASE-13071 Project: HBase Issue Type: New Feature Affects Versions: 0.98.11 Reporter: Eshcar Hillel Attachments: HBASE-13071-v1.patch, HBaseStreamingScanDesign.pdf, HbaseStreamingScanEvaluation.pdf A scan operation iterates over all rows of a table or a subrange of the table. The synchronous nature in which the data is served at the client side hinders the speed the application traverses the data: it increases the overall processing time, and may cause a great variance in the times the application waits for the next piece of data. The scanner next() method at the client side invokes an RPC to the regionserver and then stores the results in a cache. The application can specify how many rows will be transmitted per RPC; by default this is set to 100 rows. The cache can be considered as a producer-consumer queue, where the hbase client pushes the data to the queue and the application consumes it. Currently this queue is synchronous, i.e., blocking. More specifically, when the application consumed all the data from the cache --- so the cache is empty --- the hbase client retrieves additional data from the server and re-fills the cache with new data. During this time the application is blocked. Under the assumption that the application processing time can be balanced by the time it takes to retrieve the data, an asynchronous approach can reduce the time the application is waiting for data. We attach a design document. We also have a patch that is based on a private branch, and some evaluation results of this code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13071) Hbase Streaming Scan Feature
[ https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-13071: -- Attachment: HBASE-13071-v2.patch Hbase Streaming Scan Feature Key: HBASE-13071 URL: https://issues.apache.org/jira/browse/HBASE-13071 Project: HBase Issue Type: New Feature Affects Versions: 0.98.11 Reporter: Eshcar Hillel Attachments: HBASE-13071-v1.patch, HBASE-13071-v2.patch, HBaseStreamingScanDesign.pdf, HbaseStreamingScanEvaluation.pdf A scan operation iterates over all rows of a table or a subrange of the table. The synchronous nature in which the data is served at the client side hinders the speed the application traverses the data: it increases the overall processing time, and may cause a great variance in the times the application waits for the next piece of data. The scanner next() method at the client side invokes an RPC to the regionserver and then stores the results in a cache. The application can specify how many rows will be transmitted per RPC; by default this is set to 100 rows. The cache can be considered as a producer-consumer queue, where the hbase client pushes the data to the queue and the application consumes it. Currently this queue is synchronous, i.e., blocking. More specifically, when the application consumed all the data from the cache --- so the cache is empty --- the hbase client retrieves additional data from the server and re-fills the cache with new data. During this time the application is blocked. Under the assumption that the application processing time can be balanced by the time it takes to retrieve the data, an asynchronous approach can reduce the time the application is waiting for data. We attach a design document. We also have a patch that is based on a private branch, and some evaluation results of this code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13014) Java Tool For Region Moving
[
https://issues.apache.org/jira/browse/HBASE-13014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338342#comment-14338342
]
Hari Krishna Dara commented on HBASE-13014:
---
Some comments on {{readExcludes()}} method:
* The {{close()}} calls should be in a finally block.
* The below line doesn't need an initializer:
{code}
+ String line = new String();
{code}
* Replace the below block:
{code}
+ FileInputStream fis = new FileInputStream(f);
+ BufferedReader br = new BufferedReader(new InputStreamReader(fis));
{code}
with the below and just call {{br.close()}} (it will internally {[close()}} the
underlying readers and streams).
{code}
+ BufferedReader br = new BufferedReader(new FileReader(f));
{code}
* Support comment lines (skip lines starting with #).
* You should use {{equals()}} in the below code:
{code}
+if (line != ) {
{code}
like this:
{code}
+if (line.equals()) {
{code}
Better yet, use the {{StringUtils.isEmpty()}} from commons.
You can simplify the whole logic of {{stripServers()}}, if you just
preconstruct the {{host:port}} combination, something like this:
{code}
private String stripServer(ArrayListString regionServers, String hostname,
int port)
throws Exception {
String server = hostname+ServerName.SERVERNAME_SEPARATOR+port;
if (! regionServers.remove(server)) {
throw new Exception(Server: + server + is not Online);
}
return regionServers;
}
{code}
Couple of {{close()}} calls in {{isSuccessfulScan()}} method that are not in
{{finally}} block.
Some comments on {{getServerNameForRegion()}}
* A few {{close()}} calls not in {{finally}} block
* Returning {{null}} would be clearer here:
{code}
+ return server;
{code}
* Do you really intend to wait on {{tracker.isLocationAvailable()}} for ever?
Is there a reason for sleeping outside in {{Thread.sleep()}} instead of letting
the tracker do the wait? I recommend just using
{{tracker.waitMetaRegionLocation()}} with an upper bound on {{timeout}} and
fail the operation if a {{null}} is returned, and this method can be
interrupted too, if that is what you were looking for.
* The outer {{new String()}} is not needed here:
{code}
+ new String(Bytes.toString(servername).replaceFirst(:, ,) +
,
+ + Bytes.toLong(startcode));
{code}
Misc. others
* You have a couple of {{System.out.println()}}, you might want to replace with
{{LOG}} calls.
* The below message could get misleading, since it could catch any unchecked
exceptions also. Better to create an exception type, or may be even just use
the return value of {{stripServer()}} to signal the error condition
{code}
+ } catch (Exception e) {
+LOG.info(hostname= + hostname + is not up yet, waiting, e);
+ }
{code}
* Consider calling {{ExecutorService.shutdown()}} after adding all the tasks.
The way it is right now, we have the pool leaking (applies to both {{load()}}
and {{unload()}}).
Java Tool For Region Moving
Key: HBASE-13014
URL: https://issues.apache.org/jira/browse/HBASE-13014
Project: HBase
Issue Type: Improvement
Reporter: Abhishek Singh Chouhan
Assignee: Abhishek Singh Chouhan
Attachments: HBASE-13014.patch
As per discussion on HBASE-12989 we should move the functionality of
region_mover.rb into a Java tool and use region_mover.rb only only as a
wrapper around it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13071) Hbase Streaming Scan Feature
[ https://issues.apache.org/jira/browse/HBASE-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-13071: -- Attachment: HBASE-13071-v1.patch Hbase Streaming Scan Feature Key: HBASE-13071 URL: https://issues.apache.org/jira/browse/HBASE-13071 Project: HBase Issue Type: New Feature Reporter: Eshcar Hillel Attachments: HBASE-13071-v1.patch, HBaseStreamingScanDesign.pdf, HbaseStreamingScanEvaluation.pdf A scan operation iterates over all rows of a table or a subrange of the table. The synchronous nature in which the data is served at the client side hinders the speed the application traverses the data: it increases the overall processing time, and may cause a great variance in the times the application waits for the next piece of data. The scanner next() method at the client side invokes an RPC to the regionserver and then stores the results in a cache. The application can specify how many rows will be transmitted per RPC; by default this is set to 100 rows. The cache can be considered as a producer-consumer queue, where the hbase client pushes the data to the queue and the application consumes it. Currently this queue is synchronous, i.e., blocking. More specifically, when the application consumed all the data from the cache --- so the cache is empty --- the hbase client retrieves additional data from the server and re-fills the cache with new data. During this time the application is blocked. Under the assumption that the application processing time can be balanced by the time it takes to retrieve the data, an asynchronous approach can reduce the time the application is waiting for data. We attach a design document. We also have a patch that is based on a private branch, and some evaluation results of this code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13111) truncate_preserve command is failing with undefined method error
Ashish Singhi created HBASE-13111:
-
Summary: truncate_preserve command is failing with undefined
method error
Key: HBASE-13111
URL: https://issues.apache.org/jira/browse/HBASE-13111
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 1.0.0, 2.0.0, 1.1.0
Reporter: Ashish Singhi
Assignee: Ashish Singhi
{noformat}
hbase(main):001:0 truncate_preserve 't1'
Truncating 't1' table (it may take a while):
ERROR: undefined method `getTable' for nil:NilClass
Here is some help for this command:
Disables, drops and recreates the specified table while still maintaing the
previous region boundaries.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13112) quota.rb, security.rb and visibility_labels.rb leaks connection
[ https://issues.apache.org/jira/browse/HBASE-13112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-13112: -- Status: Patch Available (was: Open) quota.rb, security.rb and visibility_labels.rb leaks connection --- Key: HBASE-13112 URL: https://issues.apache.org/jira/browse/HBASE-13112 Project: HBase Issue Type: Bug Components: shell Affects Versions: 1.0.0, 2.0.0, 1.1.0 Reporter: Ashish Singhi Assignee: Ashish Singhi Attachments: HBASE-13112.patch quota.rb, security.rb and visibility_labels.rb leaks connection as noted by [~ndimiduk] [here|https://issues.apache.org/jira/browse/HBASE-12833?focusedCommentId=14279465page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14279465] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13099) Scans as in DynamoDB
[ https://issues.apache.org/jira/browse/HBASE-13099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338467#comment-14338467 ] Nicolas Liochon commented on HBASE-13099: - The 1mb could be changed / made configurable. The scan could finish if we are at the end of a row and one of these conditions is met: - we already have more than XX Mb and - the scan has been running for more than YY seconds - the scan reached the end of a region This could simplify some code, and make the server less sensitive to client issues. This would allow to remove the small scan code in the client as well (and, for all the clients that are doing small scans w/o setting this small flag, it would be faster). Scans as in DynamoDB Key: HBASE-13099 URL: https://issues.apache.org/jira/browse/HBASE-13099 Project: HBase Issue Type: Brainstorming Components: Client, regionserver Reporter: Nicolas Liochon cc: [~saint@gmail.com] - as discussed offline. DynamoDB has a very simple way to manage scans server side: ??citation?? The data returned from a Query or Scan operation is limited to 1 MB; this means that if you scan a table that has more than 1 MB of data, you'll need to perform another Scan operation to continue to the next 1 MB of data in the table. If you query or scan for specific attributes that match values that amount to more than 1 MB of data, you'll need to perform another Query or Scan request for the next 1 MB of data. To do this, take the LastEvaluatedKey value from the previous request, and use that value as the ExclusiveStartKey in the next request. This will let you progressively query or scan for new data in 1 MB increments. When the entire result set from a Query or Scan has been processed, the LastEvaluatedKey is null. This indicates that the result set is complete (i.e. the operation processed the “last page” of data). ??citation?? This means that there is no state server side: the work is done client side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-13100) Shell command to retrieve table splits
[
https://issues.apache.org/jira/browse/HBASE-13100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Singhi reassigned HBASE-13100:
-
Assignee: Ashish Singhi
Shell command to retrieve table splits
--
Key: HBASE-13100
URL: https://issues.apache.org/jira/browse/HBASE-13100
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Sean Busbey
Assignee: Ashish Singhi
Priority: Minor
Labels: beginner
Fix For: 2.0.0, 1.1.0
Add a shell command that returns the splits for a table.
Doing this yourself is currently possible, but involves going outside of the
public api.
{code}
jruby-1.7.3 :012 create 'example_table', 'f1', SPLITS = [10, 20, 30,
40]
0 row(s) in 0.5500 seconds
= Hbase::Table - example_table
jruby-1.7.3 :013
get_table('example_table').table.get_all_region_locations.map do |location|
org.apache.hadoop.hbase.util.Bytes::toStringBinary(location.get_region_info.get_start_key)
end
0 row(s) in 0.0130 seconds
= [, 10, 20, 30, 40]
jruby-1.7.3 :014
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13097) Reduce Connection and RpcClient creations in unit tests
[
https://issues.apache.org/jira/browse/HBASE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangduo updated HBASE-13097:
-
Attachment: HBASE-13097.patch
People challenged that we need to use different connections in tests to expose
more possible problems. That's reasonable I think, so there is another try.
Introduce a 'use global EventLoopGroup' config(so many configs :(), default is
true. If true, will use a global EventLoopGroup instead of create a new one.
Do you guys think this is OK? Thanks. [~jurmous] [~stack]
Reduce Connection and RpcClient creations in unit tests
---
Key: HBASE-13097
URL: https://issues.apache.org/jira/browse/HBASE-13097
Project: HBase
Issue Type: Bug
Components: IPC/RPC, test
Affects Versions: 2.0.0, 1.1.0
Reporter: zhangduo
Attachments: HBASE-13097.patch
In some unit tests(such as TestAcidGuarantees) we create multiple Connection
instance. If we use AsyncRpcClient, then there will be multiple netty
Bootstrap and every Bootstrap has its own PooledByteBufAllocator.
I haven't read the code clearly but it uses some threadlocal technics and
jmap shows io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry is the
biggest things on Heap.
See
https://builds.apache.org/job/HBase-TRUNK/6168/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestAcidGuarantees-output.txt
{noformat}
2015-02-24 23:50:29,704 WARN [JvmPauseMonitor]
util.JvmPauseMonitor$Monitor(167): Detected pause in JVM or host machine (eg
GC): pause of approximately 20133ms
GC pool 'PS MarkSweep' had collection(s): count=15 time=55525ms
{noformat}
Update: We use a singleton PooledByteBufAllocator so the reason should be too
many threads. So we will work on reduce the connections and rpclients in unit
tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13112) quota.rb, security.rb and visibility_labels.rb leaks connection
[ https://issues.apache.org/jira/browse/HBASE-13112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-13112: -- Attachment: HBASE-13112.patch quota.rb, security.rb and visibility_labels.rb leaks connection --- Key: HBASE-13112 URL: https://issues.apache.org/jira/browse/HBASE-13112 Project: HBase Issue Type: Bug Components: shell Affects Versions: 1.0.0, 2.0.0, 1.1.0 Reporter: Ashish Singhi Assignee: Ashish Singhi Attachments: HBASE-13112.patch quota.rb, security.rb and visibility_labels.rb leaks connection as noted by [~ndimiduk] [here|https://issues.apache.org/jira/browse/HBASE-12833?focusedCommentId=14279465page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14279465] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13114) [UNITTEST] TestEnableTableHandler.testDeleteForSureClearsAllTableRowsFromMeta
stack created HBASE-13114:
-
Summary: [UNITTEST]
TestEnableTableHandler.testDeleteForSureClearsAllTableRowsFromMeta
Key: HBASE-13114
URL: https://issues.apache.org/jira/browse/HBASE-13114
Project: HBase
Issue Type: Bug
Components: test
Reporter: stack
Assignee: stack
I've seen this fail a few times. It just happened now on internal rig. Looking
into it
{code}
REGRESSION:
org.apache.hadoop.hbase.master.handler.TestEnableTableHandler.testDeleteForSureClearsAllTableRowsFromMeta
Error Message:
expected:0 but was:1
Stack Trace:
java.lang.AssertionError: expected:0 but was:1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hbase.master.handler.TestEnableTableHandler.testDeleteForSureClearsAllTableRowsFromMeta(TestEnableTableHandler.java:151)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12972) Region, a supportable public/evolving subset of HRegion
[
https://issues.apache.org/jira/browse/HBASE-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338776#comment-14338776
]
Sean Busbey commented on HBASE-12972:
-
{quote}
bq. 'm treating introduction of the Region interface as a singularity of sorts
for coprocessors: neither source nor binary compatibility will be maintained. I
don't see the harm in a singularity, HRegion isn't supported, that's the point
of this work... to replace it with something that is. However, after there is a
first workable patch if it's not too onerous to make addtional changes that
keep source or binary compatibility then we can do that.
Let's discuss.
{quote}
We just started having a compatibility statement, so I'd really like to avoid
breaking it right away. What's the timeline for Phoenix 1.0? Do we have time to
get a HBase 1.1.0 out?
I haven't seen a discussion of how we interpret the add new feature part of
semver, so my apologies if I've missed it. My own interpretation is that the
addition of an API like Region qualifies, since a user that built on it
wouldn't be able to go back to the 1.0.0 release without things breaking.
What if we marked Region LimitedPrivate / Unstable in 1.0.z? We say that those
things may break in patch releases, so someone following our compatibility
guidelines wouldn't have an expectation that they can rely on it and still go
back to 1.0.0. We could mark it Evolving (or Stable or whatever you please) in
1.1.0+.
With that change, we could do similar overloading and runtime checking to that
done in HBASE-10378 to make sure we keep our compat promises.
Region, a supportable public/evolving subset of HRegion
---
Key: HBASE-12972
URL: https://issues.apache.org/jira/browse/HBASE-12972
Project: HBase
Issue Type: New Feature
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.11
Attachments: HBASE-12972-0.98.patch, HBASE-12972-0.98.patch,
HBASE-12972-0.98.patch
On HBASE-12566, [~lhofhansl] proposed:
{quote}
Maybe we can have a {{Region}} interface that is to {{HRegion}} is what
{{Store}} is to {{HStore}}. Store marked with {{@InterfaceAudience.Private}}
but used in some coprocessor hooks.
{quote}
By example, now coprocessors have to reach into HRegion in order to
participate in row and region locking protocols, this is one area where the
functionality is legitimate for coprocessors but not for users, so an
in-between interface make sense.
In addition we should promote {{Store}}'s interface audience to
LimitedPrivate(COPROC).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13114) [UNITTEST] TestEnableTableHandler.testDeleteForSureClearsAllTableRowsFromMeta
[
https://issues.apache.org/jira/browse/HBASE-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338785#comment-14338785
]
Andrey Stepachev commented on HBASE-13114:
--
what version was it [~stack]?
[UNITTEST] TestEnableTableHandler.testDeleteForSureClearsAllTableRowsFromMeta
-
Key: HBASE-13114
URL: https://issues.apache.org/jira/browse/HBASE-13114
Project: HBase
Issue Type: Bug
Components: test
Reporter: stack
Assignee: stack
I've seen this fail a few times. It just happened now on internal rig.
Looking into it
{code}
REGRESSION:
org.apache.hadoop.hbase.master.handler.TestEnableTableHandler.testDeleteForSureClearsAllTableRowsFromMeta
Error Message:
expected:0 but was:1
Stack Trace:
java.lang.AssertionError: expected:0 but was:1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hbase.master.handler.TestEnableTableHandler.testDeleteForSureClearsAllTableRowsFromMeta(TestEnableTableHandler.java:151)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12924) HRegionServer#MovedRegionsCleaner Chore does not start
[
https://issues.apache.org/jira/browse/HBASE-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338820#comment-14338820
]
Jonathan Lawlor commented on HBASE-12924:
-
Thanks for fixing this one [~asameet], some review on patch below:
- Because the chore is started outside of MovedRegionsCleaner#createAndStart it
may be best to rename the method to simply MovedRegionsCleaner#create
- As [~stack] pointed out, there are some formatting issues in the test class.
If you're developing in eclipse there is a formatter
(dev-support/hbase_eclipse_formatter.xml) that will do all the formatting for
you every time you save
- May want to get the timeout value used for this chore in a method that can be
overriden for tests
Otherwise looks good, thanks for the patch
HRegionServer#MovedRegionsCleaner Chore does not start
--
Key: HBASE-12924
URL: https://issues.apache.org/jira/browse/HBASE-12924
Project: HBase
Issue Type: Bug
Reporter: Jonathan Lawlor
Priority: Minor
Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.11
Attachments: first-task.patch
This issue is very similar to the one described in HBASE-11354. The method
createAndStart in MovedRegionsCleaner creates an instance of the chore but
never starts the underlying thread:
{code}
static MovedRegionsCleaner createAndStart(HRegionServer rs){
Stoppable stoppable = new Stoppable() {
private volatile boolean isStopped = false;
@Override public void stop(String why) { isStopped = true;}
@Override public boolean isStopped() {return isStopped;}
};
return new MovedRegionsCleaner(rs, stoppable);
}
{code}
Nobody has noticed this issue so far so the Chore may not be that important
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12972) Region, a supportable public/evolving subset of HRegion
[
https://issues.apache.org/jira/browse/HBASE-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338793#comment-14338793
]
Andrew Purtell commented on HBASE-12972:
bq. I haven't seen a discussion of how we interpret the add new feature part
of semver, so my apologies if I've missed it. My own interpretation is that the
addition of an API like Region qualifies, since a user that built on it
wouldn't be able to go back to the 1.0.0 release without things breaking.
That sucks but is valid.
However, HRegion is Private. This is enabling something that technically
shouldn't have been done in the first place.
bq. What if we marked Region LimitedPrivate / Unstable in 1.0.z?
Then why would someone use it? Region is supposed to be the supportable version
of HRegion. Unstable provides only effectively the same guarantees as
inappropriate use of Private interfaces.
We can strongly suggest Phoenix drop its local mutable indexing feature as a
mistake until HBase can support it, in 1.1 or 2.0 or whatever, but I don't
think they will go for that. So, it's likely that use of Private interfaces
will continue without having something like Region in place on the occasion of
the port of Phoenix to HBase 1.0. If that's the case then this work is moot
now, we can table it until we are nearing a 1.1 or 2.0 release.
Region, a supportable public/evolving subset of HRegion
---
Key: HBASE-12972
URL: https://issues.apache.org/jira/browse/HBASE-12972
Project: HBase
Issue Type: New Feature
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.11
Attachments: HBASE-12972-0.98.patch, HBASE-12972-0.98.patch,
HBASE-12972-0.98.patch
On HBASE-12566, [~lhofhansl] proposed:
{quote}
Maybe we can have a {{Region}} interface that is to {{HRegion}} is what
{{Store}} is to {{HStore}}. Store marked with {{@InterfaceAudience.Private}}
but used in some coprocessor hooks.
{quote}
By example, now coprocessors have to reach into HRegion in order to
participate in row and region locking protocols, this is one area where the
functionality is legitimate for coprocessors but not for users, so an
in-between interface make sense.
In addition we should promote {{Store}}'s interface audience to
LimitedPrivate(COPROC).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12972) Region, a supportable public/evolving subset of HRegion
[
https://issues.apache.org/jira/browse/HBASE-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338812#comment-14338812
]
Andrew Purtell commented on HBASE-12972:
Also, we haven't really gotten to the hard part yet, where we show folks like
[~rajeshbabu], [~jesse_yates], [~jamestaylor], and others on Phoenix the new
opaque SplitTransaction and RegionMergeTransaction interfaces and ask them if
local mutable indexes can be reimplemented without diving into the details. You
can see the trouble spots in Phoenix by applying the the latest patch on
PHOENIX-1681 to the Phoenix 4.0 branch, then grep for wherever the new private
implementation classes SplitTransactionImpl and RegionMergeTransactionImpl are
referenced or extended.
Region, a supportable public/evolving subset of HRegion
---
Key: HBASE-12972
URL: https://issues.apache.org/jira/browse/HBASE-12972
Project: HBase
Issue Type: New Feature
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.11
Attachments: HBASE-12972-0.98.patch, HBASE-12972-0.98.patch,
HBASE-12972-0.98.patch
On HBASE-12566, [~lhofhansl] proposed:
{quote}
Maybe we can have a {{Region}} interface that is to {{HRegion}} is what
{{Store}} is to {{HStore}}. Store marked with {{@InterfaceAudience.Private}}
but used in some coprocessor hooks.
{quote}
By example, now coprocessors have to reach into HRegion in order to
participate in row and region locking protocols, this is one area where the
functionality is legitimate for coprocessors but not for users, so an
in-between interface make sense.
In addition we should promote {{Store}}'s interface audience to
LimitedPrivate(COPROC).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13091) Split ZK Quorum on Master WebUI
[ https://issues.apache.org/jira/browse/HBASE-13091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338849#comment-14338849 ] Jean-Marc Spaggiari commented on HBASE-13091: - The logic was already there for this field, so it's less impact to just update the existing one instead of changing the max limit and see what the result is for all the fields. Split ZK Quorum on Master WebUI --- Key: HBASE-13091 URL: https://issues.apache.org/jira/browse/HBASE-13091 Project: HBase Issue Type: Bug Affects Versions: 1.0.1, 0.98.10.1 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Attachments: HBASE-13091-v0-trunk.patch, HBASE-13091-v1-trunk.patch, screenshot.png When using ZK servers or more, on the Master WebUI, this create a very large column and so reduce a lot the others, splitting all the lines and creating tall cells Splitting the ZK quorum with one per line will make it nicer and easier to read. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13111) truncate_preserve command is failing with undefined method error
[
https://issues.apache.org/jira/browse/HBASE-13111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338696#comment-14338696
]
Hudson commented on HBASE-13111:
SUCCESS: Integrated in HBase-1.0 #777 (See
[https://builds.apache.org/job/HBase-1.0/777/])
HBASE-13111 truncate_preserve command is failing with undefined method error
(Ashish Singhi) (tedyu: rev 6fbd2eec56c296638f44e05dc64aeb00e6c074b6)
* hbase-shell/src/main/ruby/hbase/admin.rb
truncate_preserve command is failing with undefined method error
Key: HBASE-13111
URL: https://issues.apache.org/jira/browse/HBASE-13111
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 1.0.0, 2.0.0, 1.1.0
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.0.1, 1.1.0
Attachments: HBASE-13111.patch
{noformat}
hbase(main):001:0 truncate_preserve 't1'
Truncating 't1' table (it may take a while):
ERROR: undefined method `getTable' for nil:NilClass
Here is some help for this command:
Disables, drops and recreates the specified table while still maintaing the
previous region boundaries.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12924) HRegionServer#MovedRegionsCleaner Chore does not start
[
https://issues.apache.org/jira/browse/HBASE-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338732#comment-14338732
]
stack commented on HBASE-12924:
---
Can we have it so we don't have test wait for 240 seconds for test to complete?
How about setting down the timers for when this task runs to a few seconds?
Fix formatting in test too so it is like other tests (two spaces for tab).
Otherwise, nice patch. Thanks.
HRegionServer#MovedRegionsCleaner Chore does not start
--
Key: HBASE-12924
URL: https://issues.apache.org/jira/browse/HBASE-12924
Project: HBase
Issue Type: Bug
Reporter: Jonathan Lawlor
Priority: Minor
Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.11
Attachments: first-task.patch
This issue is very similar to the one described in HBASE-11354. The method
createAndStart in MovedRegionsCleaner creates an instance of the chore but
never starts the underlying thread:
{code}
static MovedRegionsCleaner createAndStart(HRegionServer rs){
Stoppable stoppable = new Stoppable() {
private volatile boolean isStopped = false;
@Override public void stop(String why) { isStopped = true;}
@Override public boolean isStopped() {return isStopped;}
};
return new MovedRegionsCleaner(rs, stoppable);
}
{code}
Nobody has noticed this issue so far so the Chore may not be that important
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12972) Region, a supportable public/evolving subset of HRegion
[
https://issues.apache.org/jira/browse/HBASE-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338752#comment-14338752
]
Andrew Purtell commented on HBASE-12972:
bq. Looks like the current patch makes changes that won't allow it in 1.0.z
(non-binary compat changes to existing LimitedPrivate.Evolving, adding new
features to LimitedPrivate.Evolving).
Please point these changes out where you find them.
bq. Can we aim for 0.98.y, 1.1.z, 2.y.z, or would folks prefer feedback to make
it compat?
Well we should have a supportable 'Region' interface for Phoenix 1.0, which
implies in HBase 1.0.x at least. Otherwise they are going to need to continue
to work directly with HRegion and SplitTransaction, and other things I would
really like to -1 over there but can't if there is no realistic alternative.
I did say this above:
{quote}
'm treating introduction of the Region interface as a singularity of sorts for
coprocessors: neither source nor binary compatibility will be maintained. I
don't see the harm in a singularity, HRegion isn't supported, that's the point
of this work... to replace it with something that is. However, after there is a
first workable patch if it's not too onerous to make addtional changes that
keep source or binary compatibility then we can do that.
{quote}
Let's discuss.
Region, a supportable public/evolving subset of HRegion
---
Key: HBASE-12972
URL: https://issues.apache.org/jira/browse/HBASE-12972
Project: HBase
Issue Type: New Feature
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.11
Attachments: HBASE-12972-0.98.patch, HBASE-12972-0.98.patch,
HBASE-12972-0.98.patch
On HBASE-12566, [~lhofhansl] proposed:
{quote}
Maybe we can have a {{Region}} interface that is to {{HRegion}} is what
{{Store}} is to {{HStore}}. Store marked with {{@InterfaceAudience.Private}}
but used in some coprocessor hooks.
{quote}
By example, now coprocessors have to reach into HRegion in order to
participate in row and region locking protocols, this is one area where the
functionality is legitimate for coprocessors but not for users, so an
in-between interface make sense.
In addition we should promote {{Store}}'s interface audience to
LimitedPrivate(COPROC).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11544) [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME
[ https://issues.apache.org/jira/browse/HBASE-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Lawlor updated HBASE-11544: Attachment: HBASE-11544-v6.patch Forgot to include these javadoc fixes in last patch [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME -- Key: HBASE-11544 URL: https://issues.apache.org/jira/browse/HBASE-11544 Project: HBase Issue Type: Bug Reporter: stack Assignee: Jonathan Lawlor Priority: Critical Labels: beginner Attachments: HBASE-11544-v1.patch, HBASE-11544-v2.patch, HBASE-11544-v3.patch, HBASE-11544-v4.patch, HBASE-11544-v5.patch, HBASE-11544-v6.patch, HBASE-11544-v6.patch Running some tests, I set hbase.client.scanner.caching=1000. Dataset has large cells. I kept OOME'ing. Serverside, we should measure how much we've accumulated and return to the client whatever we've gathered once we pass out a certain size threshold rather than keep accumulating till we OOME. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11544) [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME
[ https://issues.apache.org/jira/browse/HBASE-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Lawlor updated HBASE-11544: Status: Open (was: Patch Available) [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME -- Key: HBASE-11544 URL: https://issues.apache.org/jira/browse/HBASE-11544 Project: HBase Issue Type: Bug Reporter: stack Assignee: Jonathan Lawlor Priority: Critical Labels: beginner Attachments: HBASE-11544-v1.patch, HBASE-11544-v2.patch, HBASE-11544-v3.patch, HBASE-11544-v4.patch, HBASE-11544-v5.patch, HBASE-11544-v6.patch, HBASE-11544-v6.patch Running some tests, I set hbase.client.scanner.caching=1000. Dataset has large cells. I kept OOME'ing. Serverside, we should measure how much we've accumulated and return to the client whatever we've gathered once we pass out a certain size threshold rather than keep accumulating till we OOME. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
