[jira] [Commented] (HBASE-12116) Hot contention spots; writing
[
https://issues.apache.org/jira/browse/HBASE-12116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493630#comment-14493630
]
stack commented on HBASE-12116:
---
Esteban and I did some messing with a YCSB pure write load this afternoon. We
got to a place where the four disks were saturated writing (CPU was not).
Here's a few notes from the session:
+ Esteban found that upping ringbuffer slot count helped; default is 16k.. he
was running at 512k (hbase.regionserver.wal.disruptor.event.count). Also said
deferred sync helped too.
+ Would be good to figure how to do bigger writes so we get more on disk per op.
+ Adding to CSLM doPut, CRC, and compares are the big consumers of CPU (apart
from G1GC) according to perf (see below), no surprise. Flight Recorder has CRC
and unsafe compareto at 20% each, then CSLM#doPut at 12%.
+ Minors but interesting (according to FR) are ByteBloomFilter#set... could do
with a tune up (the BB#gets are showing as 3.5% -- unsafe it). The murmur
hashing shows up as a percent too (look at faster implementaitons -- guava?).
We has ACL audit on. Thats 2% of CPU according to perf.
Perf top during run.
{code}
6.53% perf-18890.map [.]
Ljava/util/concurrent/ConcurrentSkipListMap;.doPut
5.78% libjvm.so [.] OtherRegionsTable::add_reference(void*, int)
4.14% perf-18890.map [.] Lorg/apache/hadoop/util/PureJavaCrc32;.update
3.27% libjvm.so [.] G1RemSet::refine_card(signed char*, int, bool)
2.99% libjvm.so [.] G1ParCopyClosurefalse, (G1Barrier)2,
false::copy_to_survivor_space(oopDesc*)
2.90% libjvm.so [.]
G1BlockOffsetArrayContigSpace::block_start_unsafe(void const*)
2.69% libjvm.so [.]
HeapRegion::oops_on_card_seq_iterate_careful(MemRegion,
FilterOutOfRegionClosure*, bool, signed char*)
2.66% libc-2.12.so[.] memcpy
2.54% perf-18890.map [.]
Lorg/apache/hadoop/hbase/KeyValue$KVComparator;.compare
2.54% libjvm.so [.] void G1ParCopyClosurefalse, (G1Barrier)2,
false::do_oop_workunsigned int(unsigned int*)
2.09% perf-18890.map [.] jbyte_disjoint_arraycopy
1.84% libjvm.so [.]
instanceKlass::oop_oop_iterate_backwards_nv(oopDesc*, G1ParScanClosure*)
1.71% libjvm.so [.] G1ParScanThreadState::trim_queue()
1.63% libjvm.so [.] SparsePRT::add_card(int, int)
1.43% libjvm.so [.]
G1BlockOffsetArray::forward_to_block_containing_addr_const(HeapWord*,
HeapWord*, void const*) const
1.39% perf-18890.map [.]
Lorg/apache/hadoop/hbase/util/Bytes$LexicographicalComparerHolder$UnsafeComparer;.compareTo
1.35% libc-2.12.so[.] vfprintf
1.23% perf-18890.map [.]
Lorg/apache/hadoop/fs/FSDataOutputStream$PositionCache;.write
1.12% libjvm.so [.] G1UpdateRSOrPushRefOopClosure::do_oop(unsigned
int*)
1.09% libjvm.so [.] instanceKlass::oop_oop_iterate_nv(oopDesc*,
FilterOutOfRegionClosure*)
1.09% libjvm.so [.]
G1SATBCardTableModRefBS::mark_card_deferred(unsigned long)
1.04% libjvm.so [.]
HeapRegionDCTOC::walk_mem_region_with_cl(MemRegion, HeapWord*, HeapWord*,
OopClosure*)
{code}
Hot contention spots; writing
-
Key: HBASE-12116
URL: https://issues.apache.org/jira/browse/HBASE-12116
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Attachments: 12116.checkForReplicas.txt,
12116.stringify.and.cache.scanner.maxsize.txt, 12116.txt, Screen Shot
2014-09-29 at 5.12.51 PM.png, Screen Shot 2014-09-30 at 10.39.34 PM.png,
Screen Shot 2015-04-13 at 2.03.05 PM.png, perf.write3.svg, perf.write4.svg
Playing with flight recorder, here are some write-time contentious
synchronizations/locks (picture coming)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13307) Making methods under ScannerV2#next smaller allows them to be inlined gaining us 7% more throughput
[ https://issues.apache.org/jira/browse/HBASE-13307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-13307: -- Attachment: 13307v4.branch-1.txt Fix legit test failure. Making methods under ScannerV2#next smaller allows them to be inlined gaining us 7% more throughput --- Key: HBASE-13307 URL: https://issues.apache.org/jira/browse/HBASE-13307 Project: HBase Issue Type: Sub-task Components: Scanners Reporter: stack Assignee: stack Fix For: 2.0.0, 1.1.0 Attachments: 13307.txt, 13307v2.branch-1.txt, 13307v3.branch-1.txt, 13307v4.branch-1.txt, traces.8.svg, traces.9.svg See parent issue for patch and evidence. I was looking at graphs of our scan and found that methods were 'too big' to be inlined (looking at jvm compilation and inlining output flags -- see parent for list). Changing method size helped some. Let me commit the resultant patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13438) [branch-1] Backport Basic quota support for namespaces
[ https://issues.apache.org/jira/browse/HBASE-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493650#comment-14493650 ] Vandana Ayyalasomayajula commented on HBASE-13438: -- [~ashish singhi] -- yes that could be done, as that case is not handled in the master branch too. Please open a JIRA. Otherwise the latest patch looks good to me +1. [branch-1] Backport Basic quota support for namespaces -- Key: HBASE-13438 URL: https://issues.apache.org/jira/browse/HBASE-13438 Project: HBase Issue Type: Task Affects Versions: 1.1.0 Reporter: Ashish Singhi Assignee: Ashish Singhi Labels: quota Fix For: 1.1.0 Attachments: HBASE-13438-branch-1-v1.patch, HBASE-13438-branch-1-v2.patch, HBASE-13438-branch-1-v3.patch, HBASE-13438-branch-1.patch, HBASE-13438_withzk.patch This task for backporting HBASE-8410, HBASE-12936 and HBASE-13357 to branch-1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13447) Bypass logic in TimeRange.compare
[ https://issues.apache.org/jira/browse/HBASE-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13447: --- Resolution: Fixed Fix Version/s: 1.1.0 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the reviews Stack and Ted. Pushed to branch-1 and master. (As seen the fix version in the other subtasks of the parent Jira) Bypass logic in TimeRange.compare - Key: HBASE-13447 URL: https://issues.apache.org/jira/browse/HBASE-13447 Project: HBase Issue Type: Sub-task Components: Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Minor Fix For: 2.0.0, 1.1.0 Attachments: TimeRange.patch Bypass the 2 compare ops based on 'allTime' boolean value. When this is true just return 0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13307) Making methods under ScannerV2#next smaller allows them to be inlined gaining us 7% more throughput
[
https://issues.apache.org/jira/browse/HBASE-13307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493653#comment-14493653
]
Hadoop QA commented on HBASE-13307:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725160/13307v4.branch-1.txt
against branch-1 branch at commit 71536bdcc7d2d4b7a36a24ab9fa304cea4a8dda2.
ATTACHMENT ID: 12725160
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 javac{color}. The patch appears to cause mvn compile goal to
fail with Hadoop version 2.4.1.
Compilation errors resume:
[ERROR] COMPILATION ERROR :
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/ColumnRangeFilter.java:[119,35]
error: cannot find symbol
[ERROR] symbol: class ServerCell
[ERROR] symbol: class ServerCell
[ERROR] symbol: class ServerCell
[ERROR] symbol: class ServerCell
[ERROR] symbol: class ServerCell
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile)
on project hbase-client: Compilation failure: Compilation failure:
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/ColumnRangeFilter.java:[119,35]
error: cannot find symbol
[ERROR] symbol: class ServerCell
[ERROR] location: class ColumnRangeFilter
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/ColumnRangeFilter.java:[219,36]
error: cannot find symbol
[ERROR] symbol: class ServerCell
[ERROR] location: class ColumnRangeFilter
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/ColumnRangeFilter.java:[219,9]
error: cannot find symbol
[ERROR] symbol: class ServerCell
[ERROR] location: class ColumnRangeFilter
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java:[108,35]
error: cannot find symbol
[ERROR] symbol: class ServerCell
[ERROR] location: class ColumnPaginationFilter
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java:[147,36]
error: cannot find symbol
[ERROR] symbol: class ServerCell
[ERROR] location: class ColumnPaginationFilter
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java:[147,9]
error: cannot find symbol
[ERROR] - [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn goals -rf :hbase-client
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13693//console
This message is automatically generated.
Making methods under ScannerV2#next smaller allows them to be inlined gaining
us 7% more throughput
---
Key: HBASE-13307
URL: https://issues.apache.org/jira/browse/HBASE-13307
Project: HBase
Issue Type: Sub-task
Components: Scanners
Reporter: stack
Assignee: stack
Fix For: 2.0.0, 1.1.0
Attachments: 13307.txt, 13307v2.branch-1.txt, 13307v3.branch-1.txt,
13307v4.branch-1.txt, traces.8.svg, traces.9.svg
See parent issue for patch and evidence.
I was looking at graphs of our scan and found that methods were 'too big' to
be inlined (looking at jvm compilation and inlining output flags -- see
parent for list). Changing method size helped some. Let me commit the
resultant patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13301) Possible memory leak in BucketCache
[
https://issues.apache.org/jira/browse/HBASE-13301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493675#comment-14493675
]
Hadoop QA commented on HBASE-13301:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725141/HBASE-13301-0.98_v1.patch
against 0.98 branch at commit e9da064ccd8dd3c64d99e0863e36cdc1236779f1.
ATTACHMENT ID: 12725141
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
26 warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13690//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13690//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13690//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13690//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13690//console
This message is automatically generated.
Possible memory leak in BucketCache
---
Key: HBASE-13301
URL: https://issues.apache.org/jira/browse/HBASE-13301
Project: HBase
Issue Type: Bug
Components: BlockCache
Reporter: zhangduo
Assignee: zhangduo
Fix For: 2.0.0, 1.1.0, 0.98.13, 1.0.2
Attachments: HBASE-13301-0.98.patch, HBASE-13301-0.98_v1.patch,
HBASE-13301-branch-1.0.patch, HBASE-13301-branch-1.0.patch,
HBASE-13301-branch-1.0_v1.patch, HBASE-13301-branch-1.patch,
HBASE-13301-branch-1_v1.patch, HBASE-13301-testcase.patch,
HBASE-13301-testcase_v1.patch, HBASE-13301.patch, HBASE-13301_v1.patch,
HBASE-13301_v2.patch, HBASE-13301_v3.patch
{code:title=BucketCache.java}
public boolean evictBlock(BlockCacheKey cacheKey) {
...
if (bucketEntry.equals(backingMap.remove(cacheKey))) {
bucketAllocator.freeBlock(bucketEntry.offset());
realCacheSize.addAndGet(-1 * bucketEntry.getLength());
blocksByHFile.remove(cacheKey.getHfileName(), cacheKey);
if (removedBlock == null) {
this.blockNumber.decrementAndGet();
}
} else {
return false;
}
...
{code}
I think the problem is here. We remove a BucketEntry that should not be
removed by us, but we do not put it back and also do not do any clean up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11425) Cell/DBB end-to-end on the read-path
[ https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493712#comment-14493712 ] stack commented on HBASE-11425: --- Took another read of the doc (and above comments on it). # (Continuing from comments above), suggest adding to the doc estimation of how many extra objects will be made going this route and Vladimir's grid to show what you are focused on. # Did you fellas have a look at how others do offheaping or if there were libs you could have made use of? Would have been good to include notes on your findings in here. # The section on hasArray (if hasArray is false, it seems to imply hasByteBuffer is true) and the discussion of added APIs and when they come into effect and when they throw unsupported exceptions will need a rewrite in light of feedback above and review of recent patches (API method names I think we've cleaned up too). # Sounds like you fellas looked at netty ByteBuf too. Add in your findings I'd say. # Would have liked to have more detail around the RPC findings. You think it could be different now we have buffer reuse? We could save making a cellblock? # Looking at diagrams for perf, I can't tell if more is better or not. Suggest you write up a summary of what the diagrams are showing. # This feature when on, will be for whole server, right? Can't do by table or region, right? Thanks lads. Cell/DBB end-to-end on the read-path Key: HBASE-11425 URL: https://issues.apache.org/jira/browse/HBASE-11425 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Affects Versions: 0.99.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: HBASE-11425-E2E-NotComplete.patch, HBASE-11425.patch, Offheap reads in HBase using BBs_V2.pdf, Offheap reads in HBase using BBs_final.pdf Umbrella jira to make sure we can have blocks cached in offheap backed cache. In the entire read path, we can refer to this offheap buffer and avoid onheap copying. The high level items I can identify as of now are 1. Avoid the array() call on BB in read path.. (This is there in many classes. We can handle class by class) 2. Support Buffer based getter APIs in cell. In read path we will create a new Cell with backed by BB. Will need in CellComparator, Filter (like SCVF), CPs etc. 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy. 4. Remove all CP hooks (which are already deprecated) which deal with KVs. (In read path) Will add subtasks under this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13460) Revise the MetaLookupPool executor-related defaults (introduced in HBASE-13036)
[
https://issues.apache.org/jira/browse/HBASE-13460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493711#comment-14493711
]
Hadoop QA commented on HBASE-13460:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725145/13460-master-2.txt
against master branch at commit e9da064ccd8dd3c64d99e0863e36cdc1236779f1.
ATTACHMENT ID: 12725145
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13691//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13691//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13691//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13691//console
This message is automatically generated.
Revise the MetaLookupPool executor-related defaults (introduced in
HBASE-13036)
---
Key: HBASE-13460
URL: https://issues.apache.org/jira/browse/HBASE-13460
Project: HBase
Issue Type: Bug
Reporter: Devaraj Das
Assignee: Devaraj Das
Fix For: 1.1.0
Attachments: 13460-master-1.txt, 13460-master-2.txt,
13460-master-3.txt
The value of the max threads, max tasks, etc. can be improved to not
unnecessarily reject tasks. In one of our internal tests, we saw tasks
getting rejected (with the defaults that is currently there in the unpatched
code):
{noformat}
2015-04-09
12:36:34,111|beaver.machine|INFO|15134|140028818736896|MainThread|Caused by:
java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException:
Task
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@6658c97b
rejected from
java.util.concurrent.ThreadPoolExecutor@5a06[Running, pool size = 3,
active threads = 3, queued tasks = 300, completed tasks = 0]
2015-04-09
12:36:34,111|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208)
2015-04-09
12:36:34,112|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:145)
2015-04-09
12:36:34,112|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1246)
2015-04-09
12:36:34,112|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1152)
2015-04-09
12:36:34,113|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:370)
2015-04-09
12:36:34,113|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:321)
2015-04-09
12:36:34,113|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206)
2015-04-09
12:36:34,113|beaver.machine|INFO|15134|140028818736896|MainThread|at
[jira] [Updated] (HBASE-13450) Purge RawBytescomparator from the writers and readers after HBASE-10800
[ https://issues.apache.org/jira/browse/HBASE-13450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-13450: --- Attachment: HBASE-13450.patch Patch that removes the code in the FFT that deals with the legacyClassName ie. while setting the class name we would assign the actual class name only but for the BC cases we will support the legacyClassName. Also no where the KeyValue.RAWCOMPARATOR would be written to the FFT. Still only after HBASE-10800 we could remove the KeyValue.RAWCOMPARATOR completely from the code. Waiting for that to be commited. Purge RawBytescomparator from the writers and readers after HBASE-10800 --- Key: HBASE-13450 URL: https://issues.apache.org/jira/browse/HBASE-13450 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-13450.patch The RawBytesComparator should not be coming in the HFiles, the writers and readers as we will be trying to use the Bytes.RAW_COMPARATOR directly. Also RawBytescomparator can no longer by CellComparator as we would like to deal with CellComparator for all the comparisons going forward. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13450) Purge RawBytescomparator from the writers and readers after HBASE-10800
[ https://issues.apache.org/jira/browse/HBASE-13450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-13450: --- Status: Patch Available (was: Open) Purge RawBytescomparator from the writers and readers after HBASE-10800 --- Key: HBASE-13450 URL: https://issues.apache.org/jira/browse/HBASE-13450 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-13450.patch The RawBytesComparator should not be coming in the HFiles, the writers and readers as we will be trying to use the Bytes.RAW_COMPARATOR directly. Also RawBytescomparator can no longer by CellComparator as we would like to deal with CellComparator for all the comparisons going forward. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13301) Possible memory leak in BucketCache
[
https://issues.apache.org/jira/browse/HBASE-13301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493843#comment-14493843
]
zhangduo commented on HBASE-13301:
--
Seems the second time is fine. Let me commit.
Possible memory leak in BucketCache
---
Key: HBASE-13301
URL: https://issues.apache.org/jira/browse/HBASE-13301
Project: HBase
Issue Type: Bug
Components: BlockCache
Reporter: zhangduo
Assignee: zhangduo
Fix For: 2.0.0, 1.1.0, 0.98.13, 1.0.2
Attachments: HBASE-13301-0.98.patch, HBASE-13301-0.98_v1.patch,
HBASE-13301-branch-1.0.patch, HBASE-13301-branch-1.0.patch,
HBASE-13301-branch-1.0_v1.patch, HBASE-13301-branch-1.patch,
HBASE-13301-branch-1_v1.patch, HBASE-13301-testcase.patch,
HBASE-13301-testcase_v1.patch, HBASE-13301.patch, HBASE-13301_v1.patch,
HBASE-13301_v2.patch, HBASE-13301_v3.patch
{code:title=BucketCache.java}
public boolean evictBlock(BlockCacheKey cacheKey) {
...
if (bucketEntry.equals(backingMap.remove(cacheKey))) {
bucketAllocator.freeBlock(bucketEntry.offset());
realCacheSize.addAndGet(-1 * bucketEntry.getLength());
blocksByHFile.remove(cacheKey.getHfileName(), cacheKey);
if (removedBlock == null) {
this.blockNumber.decrementAndGet();
}
} else {
return false;
}
...
{code}
I think the problem is here. We remove a BucketEntry that should not be
removed by us, but we do not put it back and also do not do any clean up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13461) RegionSever Hlog flush BLOCKED on hbase-0.96.2-hadoop2
[ https://issues.apache.org/jira/browse/HBASE-13461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493812#comment-14493812 ] Liang Xie commented on HBASE-13461: --- Usually, it's not a hbase issue. Most of times, waitForAckedSeqno means the datanode write pipeline is somehow slow, maybe there's a slow write io or datanode internal lock. RegionSever Hlog flush BLOCKED on hbase-0.96.2-hadoop2 Key: HBASE-13461 URL: https://issues.apache.org/jira/browse/HBASE-13461 Project: HBase Issue Type: Bug Affects Versions: 0.96.2 Environment: hbase-0.96.2-hadoop2 hadoop2.2.0 Reporter: zhangjg I try to dump thread stack below: RpcServer.handler=63,port=60020 daemon prio=10 tid=0x7fdcddc5d000 nid=0x5f9 waiting for monitor entry [0x7fd289194000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98) - waiting to lock 0x7fd36c023728 (a org.apache.hadoop.hdfs.DFSOutputStream) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:59) at java.io.DataOutputStream.write(DataOutputStream.java:90) - locked 0x7fd510cfdc28 (a org.apache.hadoop.hdfs.client.HdfsDataOutputStream) at com.google.protobuf.CodedOutputStream.refreshBuffer(CodedOutputStream.java:833) at com.google.protobuf.CodedOutputStream.flush(CodedOutputStream.java:843) at com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:91) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.append(ProtobufLogWriter.java:87) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$LogSyncer.hlogFlush(FSHLog.java:1026) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1075) - locked 0x7fd2d9bbfad0 (a java.lang.Object) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240) at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1889) RpcServer.handler=12,port=60020 daemon prio=10 tid=0x7fdcddf2c800 nid=0x5c6 in Object.wait() [0x7fd28c4c7000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:1803) - locked 0x7fd45857c540 (a java.util.LinkedList) at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1697) at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1590) at org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1575) at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:121) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:135) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1098) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240) at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1889)
[jira] [Commented] (HBASE-13347) RowCounter using special filter is broken
[
https://issues.apache.org/jira/browse/HBASE-13347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493718#comment-14493718
]
Abhishek Singh Chouhan commented on HBASE-13347:
I was trying to figure out a case where this would give a wrong result.
As per the API doc Only Filter.filterKeyValue(Cell) is called AFTER all tests
for ttl, column match, deletes and max versions have been run.
Since we match the column before applying filter we would get correct row count
irrespective of the filter we use(FirstKeyOnlyFilter or
FirstKeyValueMatchingQualifiersFilter) although we might get the KV which does
not have the column we specified as that might not be the first column in the
KVs for that row.But for row count that would not matter?
Also there is a test for row counter testRowCounterHiddenColumn that is
passing with the current code(giving the correct count).
I'm new to HBase codebase so might be interpreting things wrongly :)
RowCounter using special filter is broken
-
Key: HBASE-13347
URL: https://issues.apache.org/jira/browse/HBASE-13347
Project: HBase
Issue Type: Bug
Components: mapreduce
Affects Versions: 1.0.0
Reporter: Lars George
Fix For: 2.0.0, 1.1.0, 1.0.2
The {{RowCounter}} in the {{mapreduce}} package is supposed to check if the
row count scan has a column selection added to it, and if so, use a different
filter that finds the row and counts it. But the {{qualifier.add()}} call is
missing in the {{for}} loop. See
https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/RowCounter.java#L165
Needs fixing or row count might be wrong when using {{--range}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13447) Bypass logic in TimeRange.compare
[ https://issues.apache.org/jira/browse/HBASE-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493826#comment-14493826 ] Hudson commented on HBASE-13447: FAILURE: Integrated in HBase-TRUNK #6374 (See [https://builds.apache.org/job/HBase-TRUNK/6374/]) HBASE-13447 Bypass logic in TimeRange.compare. (anoopsamjohn: rev 71536bdcc7d2d4b7a36a24ab9fa304cea4a8dda2) * hbase-common/src/main/java/org/apache/hadoop/hbase/io/TimeRange.java Bypass logic in TimeRange.compare - Key: HBASE-13447 URL: https://issues.apache.org/jira/browse/HBASE-13447 Project: HBase Issue Type: Sub-task Components: Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Minor Fix For: 2.0.0, 1.1.0 Attachments: TimeRange.patch Bypass the 2 compare ops based on 'allTime' boolean value. When this is true just return 0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13301) Possible memory leak in BucketCache
[
https://issues.apache.org/jira/browse/HBASE-13301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493851#comment-14493851
]
zhangduo commented on HBASE-13301:
--
Integrated to all branches.
Thanks all you guys who help me finish this.
Possible memory leak in BucketCache
---
Key: HBASE-13301
URL: https://issues.apache.org/jira/browse/HBASE-13301
Project: HBase
Issue Type: Bug
Components: BlockCache
Reporter: zhangduo
Assignee: zhangduo
Fix For: 2.0.0, 1.1.0, 0.98.13, 1.0.2
Attachments: HBASE-13301-0.98.patch, HBASE-13301-0.98_v1.patch,
HBASE-13301-branch-1.0.patch, HBASE-13301-branch-1.0.patch,
HBASE-13301-branch-1.0_v1.patch, HBASE-13301-branch-1.patch,
HBASE-13301-branch-1_v1.patch, HBASE-13301-testcase.patch,
HBASE-13301-testcase_v1.patch, HBASE-13301.patch, HBASE-13301_v1.patch,
HBASE-13301_v2.patch, HBASE-13301_v3.patch
{code:title=BucketCache.java}
public boolean evictBlock(BlockCacheKey cacheKey) {
...
if (bucketEntry.equals(backingMap.remove(cacheKey))) {
bucketAllocator.freeBlock(bucketEntry.offset());
realCacheSize.addAndGet(-1 * bucketEntry.getLength());
blocksByHFile.remove(cacheKey.getHfileName(), cacheKey);
if (removedBlock == null) {
this.blockNumber.decrementAndGet();
}
} else {
return false;
}
...
{code}
I think the problem is here. We remove a BucketEntry that should not be
removed by us, but we do not put it back and also do not do any clean up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13301) Possible memory leak in BucketCache
[
https://issues.apache.org/jira/browse/HBASE-13301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangduo updated HBASE-13301:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Possible memory leak in BucketCache
---
Key: HBASE-13301
URL: https://issues.apache.org/jira/browse/HBASE-13301
Project: HBase
Issue Type: Bug
Components: BlockCache
Reporter: zhangduo
Assignee: zhangduo
Fix For: 2.0.0, 1.1.0, 0.98.13, 1.0.2
Attachments: HBASE-13301-0.98.patch, HBASE-13301-0.98_v1.patch,
HBASE-13301-branch-1.0.patch, HBASE-13301-branch-1.0.patch,
HBASE-13301-branch-1.0_v1.patch, HBASE-13301-branch-1.patch,
HBASE-13301-branch-1_v1.patch, HBASE-13301-testcase.patch,
HBASE-13301-testcase_v1.patch, HBASE-13301.patch, HBASE-13301_v1.patch,
HBASE-13301_v2.patch, HBASE-13301_v3.patch
{code:title=BucketCache.java}
public boolean evictBlock(BlockCacheKey cacheKey) {
...
if (bucketEntry.equals(backingMap.remove(cacheKey))) {
bucketAllocator.freeBlock(bucketEntry.offset());
realCacheSize.addAndGet(-1 * bucketEntry.getLength());
blocksByHFile.remove(cacheKey.getHfileName(), cacheKey);
if (removedBlock == null) {
this.blockNumber.decrementAndGet();
}
} else {
return false;
}
...
{code}
I think the problem is here. We remove a BucketEntry that should not be
removed by us, but we do not put it back and also do not do any clean up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13447) Bypass logic in TimeRange.compare
[ https://issues.apache.org/jira/browse/HBASE-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493752#comment-14493752 ] Hudson commented on HBASE-13447: FAILURE: Integrated in HBase-1.1 #391 (See [https://builds.apache.org/job/HBase-1.1/391/]) HBASE-13447 Bypass logic in TimeRange.compare. (anoopsamjohn: rev ba4c14133ea931a07b5b8cff0b72c6847680883f) * hbase-common/src/main/java/org/apache/hadoop/hbase/io/TimeRange.java Bypass logic in TimeRange.compare - Key: HBASE-13447 URL: https://issues.apache.org/jira/browse/HBASE-13447 Project: HBase Issue Type: Sub-task Components: Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Minor Fix For: 2.0.0, 1.1.0 Attachments: TimeRange.patch Bypass the 2 compare ops based on 'allTime' boolean value. When this is true just return 0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13460) Revise the MetaLookupPool executor-related defaults (introduced in HBASE-13036)
[
https://issues.apache.org/jira/browse/HBASE-13460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493781#comment-14493781
]
Hadoop QA commented on HBASE-13460:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725156/13460-master-3.txt
against master branch at commit e9da064ccd8dd3c64d99e0863e36cdc1236779f1.
ATTACHMENT ID: 12725156
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13692//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13692//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13692//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13692//console
This message is automatically generated.
Revise the MetaLookupPool executor-related defaults (introduced in
HBASE-13036)
---
Key: HBASE-13460
URL: https://issues.apache.org/jira/browse/HBASE-13460
Project: HBase
Issue Type: Bug
Reporter: Devaraj Das
Assignee: Devaraj Das
Fix For: 1.1.0
Attachments: 13460-master-1.txt, 13460-master-2.txt,
13460-master-3.txt
The value of the max threads, max tasks, etc. can be improved to not
unnecessarily reject tasks. In one of our internal tests, we saw tasks
getting rejected (with the defaults that is currently there in the unpatched
code):
{noformat}
2015-04-09
12:36:34,111|beaver.machine|INFO|15134|140028818736896|MainThread|Caused by:
java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException:
Task
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@6658c97b
rejected from
java.util.concurrent.ThreadPoolExecutor@5a06[Running, pool size = 3,
active threads = 3, queued tasks = 300, completed tasks = 0]
2015-04-09
12:36:34,111|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208)
2015-04-09
12:36:34,112|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:145)
2015-04-09
12:36:34,112|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1246)
2015-04-09
12:36:34,112|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1152)
2015-04-09
12:36:34,113|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:370)
2015-04-09
12:36:34,113|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:321)
2015-04-09
12:36:34,113|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206)
2015-04-09
12:36:34,113|beaver.machine|INFO|15134|140028818736896|MainThread|at
[jira] [Commented] (HBASE-11425) Cell/DBB end-to-end on the read-path
[ https://issues.apache.org/jira/browse/HBASE-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493832#comment-14493832 ] ramkrishna.s.vasudevan commented on HBASE-11425: We will update all the other things as you have said in the doc. bq.You think it could be different now we have buffer reuse? We could save making a cellblock? We can try once again with buffers reuse. But still writing multiple cells individually to the socket was the time taking factor which was reduced when we were creating a cell block. Will come back to the other comments shortly. Thanks [~saint@gmail.com]. Cell/DBB end-to-end on the read-path Key: HBASE-11425 URL: https://issues.apache.org/jira/browse/HBASE-11425 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Affects Versions: 0.99.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: HBASE-11425-E2E-NotComplete.patch, HBASE-11425.patch, Offheap reads in HBase using BBs_V2.pdf, Offheap reads in HBase using BBs_final.pdf Umbrella jira to make sure we can have blocks cached in offheap backed cache. In the entire read path, we can refer to this offheap buffer and avoid onheap copying. The high level items I can identify as of now are 1. Avoid the array() call on BB in read path.. (This is there in many classes. We can handle class by class) 2. Support Buffer based getter APIs in cell. In read path we will create a new Cell with backed by BB. Will need in CellComparator, Filter (like SCVF), CPs etc. 3. Avoid KeyValue.ensureKeyValue() calls in read path - This make byte copy. 4. Remove all CP hooks (which are already deprecated) which deal with KVs. (In read path) Will add subtasks under this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13301) Possible memory leak in BucketCache
[
https://issues.apache.org/jira/browse/HBASE-13301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494024#comment-14494024
]
Hudson commented on HBASE-13301:
SUCCESS: Integrated in HBase-1.1 #392 (See
[https://builds.apache.org/job/HBase-1.1/392/])
HBASE-13301 Possible memory leak in BucketCache (zhangduo: rev
bcd5c4d1371e32f8126349d671c7b82582311a84)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketCache.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/IdLock.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/CachedEntryQueue.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
Possible memory leak in BucketCache
---
Key: HBASE-13301
URL: https://issues.apache.org/jira/browse/HBASE-13301
Project: HBase
Issue Type: Bug
Components: BlockCache
Reporter: zhangduo
Assignee: zhangduo
Fix For: 2.0.0, 1.1.0, 0.98.13, 1.0.2
Attachments: HBASE-13301-0.98.patch, HBASE-13301-0.98_v1.patch,
HBASE-13301-branch-1.0.patch, HBASE-13301-branch-1.0.patch,
HBASE-13301-branch-1.0_v1.patch, HBASE-13301-branch-1.patch,
HBASE-13301-branch-1_v1.patch, HBASE-13301-testcase.patch,
HBASE-13301-testcase_v1.patch, HBASE-13301.patch, HBASE-13301_v1.patch,
HBASE-13301_v2.patch, HBASE-13301_v3.patch
{code:title=BucketCache.java}
public boolean evictBlock(BlockCacheKey cacheKey) {
...
if (bucketEntry.equals(backingMap.remove(cacheKey))) {
bucketAllocator.freeBlock(bucketEntry.offset());
realCacheSize.addAndGet(-1 * bucketEntry.getLength());
blocksByHFile.remove(cacheKey.getHfileName(), cacheKey);
if (removedBlock == null) {
this.blockNumber.decrementAndGet();
}
} else {
return false;
}
...
{code}
I think the problem is here. We remove a BucketEntry that should not be
removed by us, but we do not put it back and also do not do any clean up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13438) [branch-1] Backport Basic quota support for namespaces
[ https://issues.apache.org/jira/browse/HBASE-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494027#comment-14494027 ] Ashish Singhi commented on HBASE-13438: --- Thanks Ted [branch-1] Backport Basic quota support for namespaces -- Key: HBASE-13438 URL: https://issues.apache.org/jira/browse/HBASE-13438 Project: HBase Issue Type: Task Affects Versions: 1.1.0 Reporter: Ashish Singhi Assignee: Ashish Singhi Labels: quota Fix For: 1.1.0 Attachments: HBASE-13438-branch-1-v1.patch, HBASE-13438-branch-1-v2.patch, HBASE-13438-branch-1-v3.patch, HBASE-13438-branch-1.patch, HBASE-13438_withzk.patch This task for backporting HBASE-8410, HBASE-12936 and HBASE-13357 to branch-1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13463) Quota needs to be updated in case of reverted merge
[ https://issues.apache.org/jira/browse/HBASE-13463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13463: --- Status: Patch Available (was: Open) Quota needs to be updated in case of reverted merge --- Key: HBASE-13463 URL: https://issues.apache.org/jira/browse/HBASE-13463 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Attachments: 13463-v1.txt During review of HBASE-13438, Vandana pointed out the following: in the case of MERGE_REVERTED, the assignment manager is not updating the quota. You need to add another hook in RegionStateListener class just like split. This issue is to fix the above case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13438) [branch-1] Backport Basic quota support for namespaces
[ https://issues.apache.org/jira/browse/HBASE-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494036#comment-14494036 ] Ashish Singhi commented on HBASE-13438: --- bq. I realized that in the case of MERGE_REVERTED, the assignment manager is not updating the quota [~avandana], on further investigating the code. The current code looks fine. We are decreasing the region count in namespace quota only on successful merge. So we need not handle MERGE_REVERTED until and unless user explicitly call this which does not seems to me reasonable. [branch-1] Backport Basic quota support for namespaces -- Key: HBASE-13438 URL: https://issues.apache.org/jira/browse/HBASE-13438 Project: HBase Issue Type: Task Affects Versions: 1.1.0 Reporter: Ashish Singhi Assignee: Ashish Singhi Labels: quota Fix For: 1.1.0 Attachments: HBASE-13438-branch-1-v1.patch, HBASE-13438-branch-1-v2.patch, HBASE-13438-branch-1-v3.patch, HBASE-13438-branch-1.patch, HBASE-13438_withzk.patch This task for backporting HBASE-8410, HBASE-12936 and HBASE-13357 to branch-1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13463) Quota needs to be updated in case of reverted merge
[ https://issues.apache.org/jira/browse/HBASE-13463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13463: --- Attachment: 13463-v1.txt Quota needs to be updated in case of reverted merge --- Key: HBASE-13463 URL: https://issues.apache.org/jira/browse/HBASE-13463 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Attachments: 13463-v1.txt During review of HBASE-13438, Vandana pointed out the following: in the case of MERGE_REVERTED, the assignment manager is not updating the quota. You need to add another hook in RegionStateListener class just like split. This issue is to fix the above case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13438) [branch-1] Backport Basic quota support for namespaces
[ https://issues.apache.org/jira/browse/HBASE-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493865#comment-14493865 ] Ted Yu commented on HBASE-13438: Planning to integrate to branch-1 if there is no further comment. [branch-1] Backport Basic quota support for namespaces -- Key: HBASE-13438 URL: https://issues.apache.org/jira/browse/HBASE-13438 Project: HBase Issue Type: Task Affects Versions: 1.1.0 Reporter: Ashish Singhi Assignee: Ashish Singhi Labels: quota Fix For: 1.1.0 Attachments: HBASE-13438-branch-1-v1.patch, HBASE-13438-branch-1-v2.patch, HBASE-13438-branch-1-v3.patch, HBASE-13438-branch-1.patch, HBASE-13438_withzk.patch This task for backporting HBASE-8410, HBASE-12936 and HBASE-13357 to branch-1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13450) Purge RawBytescomparator from the writers and readers after HBASE-10800
[
https://issues.apache.org/jira/browse/HBASE-13450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493875#comment-14493875
]
Hadoop QA commented on HBASE-13450:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725185/HBASE-13450.patch
against master branch at commit 71536bdcc7d2d4b7a36a24ab9fa304cea4a8dda2.
ATTACHMENT ID: 12725185
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1914 checkstyle errors (more than the master's current 1912 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestScanWithBloomError
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13694//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13694//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13694//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13694//console
This message is automatically generated.
Purge RawBytescomparator from the writers and readers after HBASE-10800
---
Key: HBASE-13450
URL: https://issues.apache.org/jira/browse/HBASE-13450
Project: HBase
Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 2.0.0
Attachments: HBASE-13450.patch
The RawBytesComparator should not be coming in the HFiles, the writers and
readers as we will be trying to use the Bytes.RAW_COMPARATOR directly. Also
RawBytescomparator can no longer by CellComparator as we would like to deal
with CellComparator for all the comparisons going forward.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13417) batchCoprocessorService() does not handle NULL keys
[
https://issues.apache.org/jira/browse/HBASE-13417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Singh Chouhan updated HBASE-13417:
---
Attachment: HBASE-13417.patch
Simple patch to use HConstants.EMPTY_START_ROW and HConstants.EMPTY_END_ROW in
case of null parameters. [~larsgeorge] Does this look ok?
batchCoprocessorService() does not handle NULL keys
---
Key: HBASE-13417
URL: https://issues.apache.org/jira/browse/HBASE-13417
Project: HBase
Issue Type: Bug
Components: Coprocessors
Affects Versions: 1.0.0
Reporter: Lars George
Priority: Minor
Labels: noob
Fix For: 2.0.0, 1.1.0, 1.0.2
Attachments: HBASE-13417.patch
The JavaDoc for {{batchCoprocessorService()}} reads:
{noformat}
* @param startKey
* start region selection with region containing this row. If {@code
null}, the
* selection will start with the first table region.
* @param endKey
* select regions up to and including the region containing this row.
If {@code null},
* selection will continue through the last table region.
{noformat}
Setting the call to {{null}} keys like so
{code}
Mapbyte[], CountResponse results = table.batchCoprocessorService(
RowCountService.getDescriptor().findMethodByName(getRowCount),
request, null, null, CountResponse.getDefaultInstance());
{code}
yields an exception:
{noformat}
java.lang.NullPointerException
at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:1187)
at
org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:744)
at
org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:723)
at
org.apache.hadoop.hbase.client.HTable.batchCoprocessorService(HTable.java:1801)
at
org.apache.hadoop.hbase.client.HTable.batchCoprocessorService(HTable.java:1778)
at coprocessor.EndpointBatchExample.main(EndpointBatchExample.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
{noformat}
This is caused by the call shipping off the keys to {{getStartKeysInRange()}}
as-is and that method uses {{Bytes.compareTo()}} on the {{null}} keys and
fails.
One can work around using {{HConstants.EMPTY_START_ROW,
HConstants.EMPTY_END_ROW}} instead, but that is not documented, nor standard
behavior. Need to handle {{null}} keys as advertised.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13461) RegionSever Hlog flush BLOCKED on hbase-0.96.2-hadoop2
[ https://issues.apache.org/jira/browse/HBASE-13461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangjg updated HBASE-13461: Attachment: hbase.png network flow diagrm as follows RegionSever Hlog flush BLOCKED on hbase-0.96.2-hadoop2 Key: HBASE-13461 URL: https://issues.apache.org/jira/browse/HBASE-13461 Project: HBase Issue Type: Bug Affects Versions: 0.96.2 Environment: hbase-0.96.2-hadoop2 hadoop2.2.0 Reporter: zhangjg Attachments: hbase.png, hbase.png I try to dump thread stack below: RpcServer.handler=63,port=60020 daemon prio=10 tid=0x7fdcddc5d000 nid=0x5f9 waiting for monitor entry [0x7fd289194000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98) - waiting to lock 0x7fd36c023728 (a org.apache.hadoop.hdfs.DFSOutputStream) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:59) at java.io.DataOutputStream.write(DataOutputStream.java:90) - locked 0x7fd510cfdc28 (a org.apache.hadoop.hdfs.client.HdfsDataOutputStream) at com.google.protobuf.CodedOutputStream.refreshBuffer(CodedOutputStream.java:833) at com.google.protobuf.CodedOutputStream.flush(CodedOutputStream.java:843) at com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:91) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.append(ProtobufLogWriter.java:87) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$LogSyncer.hlogFlush(FSHLog.java:1026) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1075) - locked 0x7fd2d9bbfad0 (a java.lang.Object) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240) at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1889) RpcServer.handler=12,port=60020 daemon prio=10 tid=0x7fdcddf2c800 nid=0x5c6 in Object.wait() [0x7fd28c4c7000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:1803) - locked 0x7fd45857c540 (a java.util.LinkedList) at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1697) at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1590) at org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1575) at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:121) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:135) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1098) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240) at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1889) RpcServer.handler=11,port=60020 daemon prio=10 tid=0x7fdcdd9e1000 nid=0x5c5 in Object.wait() [0x7fd28c5c8000] java.lang.Thread.State:
[jira] [Updated] (HBASE-13461) RegionSever Hlog flush BLOCKED on hbase-0.96.2-hadoop2
[ https://issues.apache.org/jira/browse/HBASE-13461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangjg updated HBASE-13461: Attachment: hbase.png RegionSever Hlog flush BLOCKED on hbase-0.96.2-hadoop2 Key: HBASE-13461 URL: https://issues.apache.org/jira/browse/HBASE-13461 Project: HBase Issue Type: Bug Affects Versions: 0.96.2 Environment: hbase-0.96.2-hadoop2 hadoop2.2.0 Reporter: zhangjg Attachments: hbase.png I try to dump thread stack below: RpcServer.handler=63,port=60020 daemon prio=10 tid=0x7fdcddc5d000 nid=0x5f9 waiting for monitor entry [0x7fd289194000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98) - waiting to lock 0x7fd36c023728 (a org.apache.hadoop.hdfs.DFSOutputStream) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:59) at java.io.DataOutputStream.write(DataOutputStream.java:90) - locked 0x7fd510cfdc28 (a org.apache.hadoop.hdfs.client.HdfsDataOutputStream) at com.google.protobuf.CodedOutputStream.refreshBuffer(CodedOutputStream.java:833) at com.google.protobuf.CodedOutputStream.flush(CodedOutputStream.java:843) at com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:91) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.append(ProtobufLogWriter.java:87) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$LogSyncer.hlogFlush(FSHLog.java:1026) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1075) - locked 0x7fd2d9bbfad0 (a java.lang.Object) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240) at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1889) RpcServer.handler=12,port=60020 daemon prio=10 tid=0x7fdcddf2c800 nid=0x5c6 in Object.wait() [0x7fd28c4c7000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:1803) - locked 0x7fd45857c540 (a java.util.LinkedList) at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1697) at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1590) at org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1575) at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:121) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:135) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1098) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240) at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1889) RpcServer.handler=11,port=60020 daemon prio=10 tid=0x7fdcdd9e1000 nid=0x5c5 in Object.wait() [0x7fd28c5c8000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at
[jira] [Updated] (HBASE-13438) [branch-1] Backport Basic quota support for namespaces
[ https://issues.apache.org/jira/browse/HBASE-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13438: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the patch, Ashish. Thanks for the review, Vandana. [branch-1] Backport Basic quota support for namespaces -- Key: HBASE-13438 URL: https://issues.apache.org/jira/browse/HBASE-13438 Project: HBase Issue Type: Task Affects Versions: 1.1.0 Reporter: Ashish Singhi Assignee: Ashish Singhi Labels: quota Fix For: 1.1.0 Attachments: HBASE-13438-branch-1-v1.patch, HBASE-13438-branch-1-v2.patch, HBASE-13438-branch-1-v3.patch, HBASE-13438-branch-1.patch, HBASE-13438_withzk.patch This task for backporting HBASE-8410, HBASE-12936 and HBASE-13357 to branch-1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13301) Possible memory leak in BucketCache
[
https://issues.apache.org/jira/browse/HBASE-13301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494015#comment-14494015
]
Hudson commented on HBASE-13301:
FAILURE: Integrated in HBase-1.0 #862 (See
[https://builds.apache.org/job/HBase-1.0/862/])
HBASE-13301 Possible memory leak in BucketCache (zhangduo: rev
b4f6ee9adeaf1d54c99615bf47d9c259eea819ac)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/IdLock.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketCache.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/CachedEntryQueue.java
Possible memory leak in BucketCache
---
Key: HBASE-13301
URL: https://issues.apache.org/jira/browse/HBASE-13301
Project: HBase
Issue Type: Bug
Components: BlockCache
Reporter: zhangduo
Assignee: zhangduo
Fix For: 2.0.0, 1.1.0, 0.98.13, 1.0.2
Attachments: HBASE-13301-0.98.patch, HBASE-13301-0.98_v1.patch,
HBASE-13301-branch-1.0.patch, HBASE-13301-branch-1.0.patch,
HBASE-13301-branch-1.0_v1.patch, HBASE-13301-branch-1.patch,
HBASE-13301-branch-1_v1.patch, HBASE-13301-testcase.patch,
HBASE-13301-testcase_v1.patch, HBASE-13301.patch, HBASE-13301_v1.patch,
HBASE-13301_v2.patch, HBASE-13301_v3.patch
{code:title=BucketCache.java}
public boolean evictBlock(BlockCacheKey cacheKey) {
...
if (bucketEntry.equals(backingMap.remove(cacheKey))) {
bucketAllocator.freeBlock(bucketEntry.offset());
realCacheSize.addAndGet(-1 * bucketEntry.getLength());
blocksByHFile.remove(cacheKey.getHfileName(), cacheKey);
if (removedBlock == null) {
this.blockNumber.decrementAndGet();
}
} else {
return false;
}
...
{code}
I think the problem is here. We remove a BucketEntry that should not be
removed by us, but we do not put it back and also do not do any clean up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13463) Quota needs to be updated in case of reverted merge
Ted Yu created HBASE-13463: -- Summary: Quota needs to be updated in case of reverted merge Key: HBASE-13463 URL: https://issues.apache.org/jira/browse/HBASE-13463 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu During review of HBASE-13438, Vandana pointed out the following: in the case of MERGE_REVERTED, the assignment manager is not updating the quota. You need to add another hook in RegionStateListener class just like split. This issue is to fix the above case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13394) Failed to recreate a table when quota is enabled
[
https://issues.apache.org/jira/browse/HBASE-13394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Singhi updated HBASE-13394:
--
Affects Version/s: 1.1.0
Fix Version/s: 1.1.0
Failed to recreate a table when quota is enabled
Key: HBASE-13394
URL: https://issues.apache.org/jira/browse/HBASE-13394
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 2.0.0, 1.1.0
Reporter: Y. SREENIVASULU REDDY
Assignee: Ashish Singhi
Labels: quota
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13394-v1.patch, HBASE-13394.patch
Steps to reproduce.
Enable quota by setting {{hbase.quota.enabled}} to true
Create a table say with name 't1', make sure the creation fails after adding
this table entry into namespace quota cache.
Now correct the failure and recreate the table 't1'. It fails with below
exception.
{noformat}
2015-04-02 14:23:53,729 | ERROR | FifoRpcScheduler.handler1-thread-23 |
Unexpected throwable object |
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2154)
java.lang.IllegalStateException: Table already in the cache t1
at
org.apache.hadoop.hbase.namespace.NamespaceTableAndRegionInfo.addTable(NamespaceTableAndRegionInfo.java:97)
at
org.apache.hadoop.hbase.namespace.NamespaceStateManager.addTable(NamespaceStateManager.java:171)
at
org.apache.hadoop.hbase.namespace.NamespaceStateManager.checkAndUpdateNamespaceTableCount(NamespaceStateManager.java:147)
at
org.apache.hadoop.hbase.namespace.NamespaceAuditor.checkQuotaToCreateTable(NamespaceAuditor.java:76)
at
org.apache.hadoop.hbase.quotas.MasterQuotaManager.checkNamespaceTableAndRegionQuota(MasterQuotaManager.java:344)
at
org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1781)
at
org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1818)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:42273)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2116)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at
org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}
P.S: Line numbers may not be in sync.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13463) Quota needs to be updated in case of reverted merge
[ https://issues.apache.org/jira/browse/HBASE-13463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13463: --- Fix Version/s: 1.1.0 2.0.0 Quota needs to be updated in case of reverted merge --- Key: HBASE-13463 URL: https://issues.apache.org/jira/browse/HBASE-13463 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Fix For: 2.0.0, 1.1.0 Attachments: 13463-v1.txt During review of HBASE-13438, Vandana pointed out the following: in the case of MERGE_REVERTED, the assignment manager is not updating the quota. You need to add another hook in RegionStateListener class just like split. This issue is to fix the above case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13463) Quota needs to be updated in case of reverted merge
[ https://issues.apache.org/jira/browse/HBASE-13463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13463: --- Status: Open (was: Patch Available) Quota needs to be updated in case of reverted merge --- Key: HBASE-13463 URL: https://issues.apache.org/jira/browse/HBASE-13463 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Fix For: 2.0.0, 1.1.0 Attachments: 13463-v1.txt During review of HBASE-13438, Vandana pointed out the following: in the case of MERGE_REVERTED, the assignment manager is not updating the quota. You need to add another hook in RegionStateListener class just like split. This issue is to fix the above case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13462) Handle deprecations for 2.0.0
Lars Francke created HBASE-13462: Summary: Handle deprecations for 2.0.0 Key: HBASE-13462 URL: https://issues.apache.org/jira/browse/HBASE-13462 Project: HBase Issue Type: Umbrella Reporter: Lars Francke This is an umbrella issue for everything that deals with handling deprecated code for the HBase 2.0.0 release. This was discussed on the [mailing list|http://apache-hbase.679495.n3.nabble.com/Removal-of-deprecated-features-td4069746.html] and agreed upon: * Everything that was already removed in the HBase 1.0.0 release will/can be removed * Everything that was deprecated in a subsequent 1.x release or in 2.0.0 will be removed in 3.0.0 and only documented as such in 2.0.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13417) batchCoprocessorService() does not handle NULL keys
[
https://issues.apache.org/jira/browse/HBASE-13417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Singh Chouhan updated HBASE-13417:
---
Status: Patch Available (was: Open)
batchCoprocessorService() does not handle NULL keys
---
Key: HBASE-13417
URL: https://issues.apache.org/jira/browse/HBASE-13417
Project: HBase
Issue Type: Bug
Components: Coprocessors
Affects Versions: 1.0.0
Reporter: Lars George
Priority: Minor
Labels: noob
Fix For: 2.0.0, 1.1.0, 1.0.2
Attachments: HBASE-13417.patch
The JavaDoc for {{batchCoprocessorService()}} reads:
{noformat}
* @param startKey
* start region selection with region containing this row. If {@code
null}, the
* selection will start with the first table region.
* @param endKey
* select regions up to and including the region containing this row.
If {@code null},
* selection will continue through the last table region.
{noformat}
Setting the call to {{null}} keys like so
{code}
Mapbyte[], CountResponse results = table.batchCoprocessorService(
RowCountService.getDescriptor().findMethodByName(getRowCount),
request, null, null, CountResponse.getDefaultInstance());
{code}
yields an exception:
{noformat}
java.lang.NullPointerException
at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:1187)
at
org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:744)
at
org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:723)
at
org.apache.hadoop.hbase.client.HTable.batchCoprocessorService(HTable.java:1801)
at
org.apache.hadoop.hbase.client.HTable.batchCoprocessorService(HTable.java:1778)
at coprocessor.EndpointBatchExample.main(EndpointBatchExample.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
{noformat}
This is caused by the call shipping off the keys to {{getStartKeysInRange()}}
as-is and that method uses {{Bytes.compareTo()}} on the {{null}} keys and
fails.
One can work around using {{HConstants.EMPTY_START_ROW,
HConstants.EMPTY_END_ROW}} instead, but that is not documented, nor standard
behavior. Need to handle {{null}} keys as advertised.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13461) RegionSever Hlog flush BLOCKED on hbase-0.96.2-hadoop2
[ https://issues.apache.org/jira/browse/HBASE-13461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493965#comment-14493965 ] zhangjg commented on HBASE-13461: - I have here a thread stack details. Can you guys help me locate the problem? Data is written into the very rules in each whole point will be blocked for 2 minutes, I turn off major compact and auto split. RegionSever Hlog flush BLOCKED on hbase-0.96.2-hadoop2 Key: HBASE-13461 URL: https://issues.apache.org/jira/browse/HBASE-13461 Project: HBase Issue Type: Bug Affects Versions: 0.96.2 Environment: hbase-0.96.2-hadoop2 hadoop2.2.0 Reporter: zhangjg I try to dump thread stack below: RpcServer.handler=63,port=60020 daemon prio=10 tid=0x7fdcddc5d000 nid=0x5f9 waiting for monitor entry [0x7fd289194000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98) - waiting to lock 0x7fd36c023728 (a org.apache.hadoop.hdfs.DFSOutputStream) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:59) at java.io.DataOutputStream.write(DataOutputStream.java:90) - locked 0x7fd510cfdc28 (a org.apache.hadoop.hdfs.client.HdfsDataOutputStream) at com.google.protobuf.CodedOutputStream.refreshBuffer(CodedOutputStream.java:833) at com.google.protobuf.CodedOutputStream.flush(CodedOutputStream.java:843) at com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:91) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.append(ProtobufLogWriter.java:87) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$LogSyncer.hlogFlush(FSHLog.java:1026) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1075) - locked 0x7fd2d9bbfad0 (a java.lang.Object) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240) at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1889) RpcServer.handler=12,port=60020 daemon prio=10 tid=0x7fdcddf2c800 nid=0x5c6 in Object.wait() [0x7fd28c4c7000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:1803) - locked 0x7fd45857c540 (a java.util.LinkedList) at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1697) at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1590) at org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1575) at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:121) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:135) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1098) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240) at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185) at
[jira] [Commented] (HBASE-13461) RegionSever Hlog flush BLOCKED on hbase-0.96.2-hadoop2
[ https://issues.apache.org/jira/browse/HBASE-13461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493906#comment-14493906 ] Esteban Gutierrez commented on HBASE-13461: --- What [~xieliang007] said. And is usually related to a bad drive or a network issue that is locking the datanode. If you further help [~kukuzidian] please use the user mailing lists from http://hbase.apache.org/mail-lists.html RegionSever Hlog flush BLOCKED on hbase-0.96.2-hadoop2 Key: HBASE-13461 URL: https://issues.apache.org/jira/browse/HBASE-13461 Project: HBase Issue Type: Bug Affects Versions: 0.96.2 Environment: hbase-0.96.2-hadoop2 hadoop2.2.0 Reporter: zhangjg I try to dump thread stack below: RpcServer.handler=63,port=60020 daemon prio=10 tid=0x7fdcddc5d000 nid=0x5f9 waiting for monitor entry [0x7fd289194000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98) - waiting to lock 0x7fd36c023728 (a org.apache.hadoop.hdfs.DFSOutputStream) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:59) at java.io.DataOutputStream.write(DataOutputStream.java:90) - locked 0x7fd510cfdc28 (a org.apache.hadoop.hdfs.client.HdfsDataOutputStream) at com.google.protobuf.CodedOutputStream.refreshBuffer(CodedOutputStream.java:833) at com.google.protobuf.CodedOutputStream.flush(CodedOutputStream.java:843) at com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:91) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.append(ProtobufLogWriter.java:87) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$LogSyncer.hlogFlush(FSHLog.java:1026) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1075) - locked 0x7fd2d9bbfad0 (a java.lang.Object) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240) at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1889) RpcServer.handler=12,port=60020 daemon prio=10 tid=0x7fdcddf2c800 nid=0x5c6 in Object.wait() [0x7fd28c4c7000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:1803) - locked 0x7fd45857c540 (a java.util.LinkedList) at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1697) at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1590) at org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1575) at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:121) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:135) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1098) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240) at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185) at
[jira] [Resolved] (HBASE-13461) RegionSever Hlog flush BLOCKED on hbase-0.96.2-hadoop2
[ https://issues.apache.org/jira/browse/HBASE-13461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez resolved HBASE-13461. --- Resolution: Invalid RegionSever Hlog flush BLOCKED on hbase-0.96.2-hadoop2 Key: HBASE-13461 URL: https://issues.apache.org/jira/browse/HBASE-13461 Project: HBase Issue Type: Bug Affects Versions: 0.96.2 Environment: hbase-0.96.2-hadoop2 hadoop2.2.0 Reporter: zhangjg I try to dump thread stack below: RpcServer.handler=63,port=60020 daemon prio=10 tid=0x7fdcddc5d000 nid=0x5f9 waiting for monitor entry [0x7fd289194000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98) - waiting to lock 0x7fd36c023728 (a org.apache.hadoop.hdfs.DFSOutputStream) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:59) at java.io.DataOutputStream.write(DataOutputStream.java:90) - locked 0x7fd510cfdc28 (a org.apache.hadoop.hdfs.client.HdfsDataOutputStream) at com.google.protobuf.CodedOutputStream.refreshBuffer(CodedOutputStream.java:833) at com.google.protobuf.CodedOutputStream.flush(CodedOutputStream.java:843) at com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:91) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.append(ProtobufLogWriter.java:87) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$LogSyncer.hlogFlush(FSHLog.java:1026) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1075) - locked 0x7fd2d9bbfad0 (a java.lang.Object) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240) at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1889) RpcServer.handler=12,port=60020 daemon prio=10 tid=0x7fdcddf2c800 nid=0x5c6 in Object.wait() [0x7fd28c4c7000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:1803) - locked 0x7fd45857c540 (a java.util.LinkedList) at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1697) at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1590) at org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1575) at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:121) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:135) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1098) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240) at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1889) RpcServer.handler=11,port=60020 daemon prio=10 tid=0x7fdcdd9e1000 nid=0x5c5 in Object.wait() [0x7fd28c5c8000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at
[jira] [Commented] (HBASE-13461) RegionSever Hlog flush BLOCKED on hbase-0.96.2-hadoop2
[ https://issues.apache.org/jira/browse/HBASE-13461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493994#comment-14493994 ] zhangjg commented on HBASE-13461: - Network flow diagram as follows: RegionSever Hlog flush BLOCKED on hbase-0.96.2-hadoop2 Key: HBASE-13461 URL: https://issues.apache.org/jira/browse/HBASE-13461 Project: HBase Issue Type: Bug Affects Versions: 0.96.2 Environment: hbase-0.96.2-hadoop2 hadoop2.2.0 Reporter: zhangjg I try to dump thread stack below: RpcServer.handler=63,port=60020 daemon prio=10 tid=0x7fdcddc5d000 nid=0x5f9 waiting for monitor entry [0x7fd289194000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98) - waiting to lock 0x7fd36c023728 (a org.apache.hadoop.hdfs.DFSOutputStream) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:59) at java.io.DataOutputStream.write(DataOutputStream.java:90) - locked 0x7fd510cfdc28 (a org.apache.hadoop.hdfs.client.HdfsDataOutputStream) at com.google.protobuf.CodedOutputStream.refreshBuffer(CodedOutputStream.java:833) at com.google.protobuf.CodedOutputStream.flush(CodedOutputStream.java:843) at com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:91) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.append(ProtobufLogWriter.java:87) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$LogSyncer.hlogFlush(FSHLog.java:1026) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1075) - locked 0x7fd2d9bbfad0 (a java.lang.Object) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240) at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1889) RpcServer.handler=12,port=60020 daemon prio=10 tid=0x7fdcddf2c800 nid=0x5c6 in Object.wait() [0x7fd28c4c7000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:1803) - locked 0x7fd45857c540 (a java.util.LinkedList) at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1697) at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1590) at org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1575) at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:121) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:135) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1098) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240) at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1889) RpcServer.handler=11,port=60020 daemon prio=10 tid=0x7fdcdd9e1000 nid=0x5c5 in Object.wait() [0x7fd28c5c8000] java.lang.Thread.State: TIMED_WAITING (on
[jira] [Updated] (HBASE-13463) Quota needs to be updated in case of reverted merge
[ https://issues.apache.org/jira/browse/HBASE-13463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13463: --- Status: Patch Available (was: Open) Quota needs to be updated in case of reverted merge --- Key: HBASE-13463 URL: https://issues.apache.org/jira/browse/HBASE-13463 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Fix For: 2.0.0, 1.1.0 Attachments: 13463-v1.txt, 13463-v2-branch-1.txt During review of HBASE-13438, Vandana pointed out the following: in the case of MERGE_REVERTED, the assignment manager is not updating the quota. You need to add another hook in RegionStateListener class just like split. This issue is to fix the above case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13463) Quota needs to be updated in case of reverted merge
[ https://issues.apache.org/jira/browse/HBASE-13463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13463: --- Fix Version/s: (was: 2.0.0) Quota needs to be updated in case of reverted merge --- Key: HBASE-13463 URL: https://issues.apache.org/jira/browse/HBASE-13463 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Fix For: 1.1.0 Attachments: 13463-v1.txt, 13463-v2-branch-1.txt During review of HBASE-13438, Vandana pointed out the following: in the case of MERGE_REVERTED, the assignment manager is not updating the quota. You need to add another hook in RegionStateListener class just like split. This issue is to fix the above case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13464) Remove deprecations for 2.0.0 - Part 1
[
https://issues.apache.org/jira/browse/HBASE-13464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494409#comment-14494409
]
Hadoop QA commented on HBASE-13464:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725239/HBASE-13464.1.patch
against master branch at commit 4f151444b58ae85b93f76254961358932e0ffb9b.
ATTACHMENT ID: 12725239
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 30 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1912 checkstyle errors (more than the master's current 1910 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13697//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13697//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13697//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13697//console
This message is automatically generated.
Remove deprecations for 2.0.0 - Part 1
--
Key: HBASE-13464
URL: https://issues.apache.org/jira/browse/HBASE-13464
Project: HBase
Issue Type: Sub-task
Reporter: Lars Francke
Assignee: Lars Francke
Attachments: HBASE-13464.1.patch
This removes deprecations for the following issues:
* HBASE-6038
* HBASE-1502
* HBASE-5453
* HBASE-5357
* HBASE-9870
* HBASE-10870
* HBASE-12363
* HBASE-9508
It introduces one new deprecation for
{{UnmodifyableHTableDescriptor#UnmodifyableHTableDescriptor()}} because that
depends on a deprecated constructor in {{HTableDescriptor}}. I've made the
constructor in question {{protected}} so both can be removed in 3.0.0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12116) Hot contention spots; writing
[ https://issues.apache.org/jira/browse/HBASE-12116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494454#comment-14494454 ] Andrew Purtell commented on HBASE-12116: bq. Chatting w/ Esteban, its an audit CP that is run in CDH Ah, well 2% for audit is still not bad. bq. On write path ditto (more awkward here even given no BB interface at write time). Yep, but if raising the floor of supported Hadoop version on trunk, maybe we could work up something together with HDFS for a 2.0 release timeframe. bq. A hbase native lib to do native CRC and compression so not tied to a particular HDFS version would be crazy? Sure, if taking on checksumming at our layer instead of pushing off to HDFS is something we still want to do. I think it still makes sense, we trade complexity in our code for a big cut in IOPS needed for reading. Native bits in HBase should be optional and have a Java only fallback, native code is a PITA for deployment, even if compiled for the correct architecture we may fail to dlopen if native symbol versioning gets in the way. Hot contention spots; writing - Key: HBASE-12116 URL: https://issues.apache.org/jira/browse/HBASE-12116 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Attachments: 12116.checkForReplicas.txt, 12116.stringify.and.cache.scanner.maxsize.txt, 12116.txt, Screen Shot 2014-09-29 at 5.12.51 PM.png, Screen Shot 2014-09-30 at 10.39.34 PM.png, Screen Shot 2015-04-13 at 2.03.05 PM.png, perf.write3.svg, perf.write4.svg Playing with flight recorder, here are some write-time contentious synchronizations/locks (picture coming) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9444) EncodedScannerV2#isSeeked does not behave as described in javadoc
[ https://issues.apache.org/jira/browse/HBASE-9444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Lawlor updated HBASE-9444: --- Resolution: Fixed Status: Resolved (was: Patch Available) Resolving as fixed as HBASE-9915 has addressed this issue. Please reopen if there are additional concerns not covered by the solution in HBASE-9915. EncodedScannerV2#isSeeked does not behave as described in javadoc - Key: HBASE-9444 URL: https://issues.apache.org/jira/browse/HBASE-9444 Project: HBase Issue Type: Bug Components: HFile Reporter: Chao Shi Priority: Minor Attachments: hbase-9444.patch I hit this when my tool is scanning HFiles using the scanner. I found isSeeked behaves different whether the HFiles are prefix-encoded or not. There is a test case in my patch that demonstrates the bug. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12116) Hot contention spots; writing
[ https://issues.apache.org/jira/browse/HBASE-12116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494162#comment-14494162 ] Andrew Purtell commented on HBASE-12116: bq. We has ACL audit on. Thats 2% of CPU according to perf. AccessController? Let's look, but 2-3% of perf is performance to design expectations, so that's actually good Hot contention spots; writing - Key: HBASE-12116 URL: https://issues.apache.org/jira/browse/HBASE-12116 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Attachments: 12116.checkForReplicas.txt, 12116.stringify.and.cache.scanner.maxsize.txt, 12116.txt, Screen Shot 2014-09-29 at 5.12.51 PM.png, Screen Shot 2014-09-30 at 10.39.34 PM.png, Screen Shot 2015-04-13 at 2.03.05 PM.png, perf.write3.svg, perf.write4.svg Playing with flight recorder, here are some write-time contentious synchronizations/locks (picture coming) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13463) Quota needs to be updated only in case of successful region merge
[ https://issues.apache.org/jira/browse/HBASE-13463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13463: --- Summary: Quota needs to be updated only in case of successful region merge (was: Quota needs to be updated in case of reverted merge) Quota needs to be updated only in case of successful region merge - Key: HBASE-13463 URL: https://issues.apache.org/jira/browse/HBASE-13463 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Fix For: 1.1.0 Attachments: 13463-v1.txt, 13463-v2-branch-1.txt During review of HBASE-13438, Vandana pointed out the following: in the case of MERGE_REVERTED, the assignment manager is not updating the quota. You need to add another hook in RegionStateListener class just like split. This issue is to fix the above case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12116) Hot contention spots; writing
[ https://issues.apache.org/jira/browse/HBASE-12116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494185#comment-14494185 ] Andrew Purtell commented on HBASE-12116: bq. CRC and unsafe compareto at 20% each Assumed this is with HBase doing its own checksumming. If we know we have DFSInputStream#read variants that accept BBs (and zero copy read under the covers), then we could move all reads over to this interface, check if Hadoop's native CRC (HADOOP-10838) is available to us, and if so run it over the contents of the BBs we get from HDFS? We'd need 2.6.0 as the floor of Hadoop support so maybe do this in trunk for 2.0? Hot contention spots; writing - Key: HBASE-12116 URL: https://issues.apache.org/jira/browse/HBASE-12116 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Attachments: 12116.checkForReplicas.txt, 12116.stringify.and.cache.scanner.maxsize.txt, 12116.txt, Screen Shot 2014-09-29 at 5.12.51 PM.png, Screen Shot 2014-09-30 at 10.39.34 PM.png, Screen Shot 2015-04-13 at 2.03.05 PM.png, perf.write3.svg, perf.write4.svg Playing with flight recorder, here are some write-time contentious synchronizations/locks (picture coming) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13466) Document deprecations in 1.x - Part 1
[ https://issues.apache.org/jira/browse/HBASE-13466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HBASE-13466: - Description: This documents deprecations for the following issues: * HBASE-6038 * HBASE-1502 * HBASE-5453 * HBASE-5357 * HBASE-9870 * HBASE-10870 * HBASE-12363 * HBASE-9508 Document deprecations in 1.x - Part 1 - Key: HBASE-13466 URL: https://issues.apache.org/jira/browse/HBASE-13466 Project: HBase Issue Type: Sub-task Reporter: Lars Francke Assignee: Lars Francke Attachments: HBASE-13466.1.patch This documents deprecations for the following issues: * HBASE-6038 * HBASE-1502 * HBASE-5453 * HBASE-5357 * HBASE-9870 * HBASE-10870 * HBASE-12363 * HBASE-9508 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13466) Document deprecations in 1.x - Part 1
[ https://issues.apache.org/jira/browse/HBASE-13466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494232#comment-14494232 ] stack commented on HBASE-13466: --- This is excellent. Will commit EOD unless objection. Document deprecations in 1.x - Part 1 - Key: HBASE-13466 URL: https://issues.apache.org/jira/browse/HBASE-13466 Project: HBase Issue Type: Sub-task Reporter: Lars Francke Assignee: Lars Francke Attachments: HBASE-13466.1.patch This documents deprecations for the following issues: * HBASE-6038 * HBASE-1502 * HBASE-5453 * HBASE-5357 * HBASE-9870 * HBASE-10870 * HBASE-12363 * HBASE-9508 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13204) Procedure v2 - client create/delete table sync
[ https://issues.apache.org/jira/browse/HBASE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-13204: Attachment: HBASE-13204-v0-branch-1.patch Procedure v2 - client create/delete table sync -- Key: HBASE-13204 URL: https://issues.apache.org/jira/browse/HBASE-13204 Project: HBase Issue Type: Sub-task Components: master Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 2.0.0 Attachments: HBASE-13204-v0-branch-1.patch, HBASE-13204-v0.patch client side, part of HBASE-12439/HBASE-13203 it uses the new procedure code to be know when the procedure is completed, and have a proper sync behavior on create/delete table. Review: https://reviews.apache.org/r/32391/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12116) Hot contention spots; writing
[ https://issues.apache.org/jira/browse/HBASE-12116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494434#comment-14494434 ] stack commented on HBASE-12116: --- [~apurtell] Chatting w/ Esteban, its an audit CP that is run in CDH. Yeah on CSLM worth digging in on -- big payback especially if can save a f few compares (in // with work to compress CSLM snapshots). On CRC, have work to do in HDFS if we want to continue doing our own CRC on read path so can read blocks that have been natively CRC'd and decompressed all offheap. On write path ditto (more awkward here even given no BB interface at write time). A hbase native lib to do native CRC and compression so not tied to a particular HDFS version would be crazy?, Or have HDFS do the checksum for us again (CPU for an extra seek -- ok if SSD). Hot contention spots; writing - Key: HBASE-12116 URL: https://issues.apache.org/jira/browse/HBASE-12116 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Attachments: 12116.checkForReplicas.txt, 12116.stringify.and.cache.scanner.maxsize.txt, 12116.txt, Screen Shot 2014-09-29 at 5.12.51 PM.png, Screen Shot 2014-09-30 at 10.39.34 PM.png, Screen Shot 2015-04-13 at 2.03.05 PM.png, perf.write3.svg, perf.write4.svg Playing with flight recorder, here are some write-time contentious synchronizations/locks (picture coming) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12116) Hot contention spots; writing
[ https://issues.apache.org/jira/browse/HBASE-12116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494226#comment-14494226 ] Andrew Purtell commented on HBASE-12116: I know you were just using YCSB as a load generator but if playing with YCSB you might find this interesting: https://github.com/apurtell/YCSB2/tree/new_hbase_client Blog post on coordinated omission corrections in this version: http://psy-lob-saw.blogspot.co.at/2015/03/fixing-ycsb-coordinated-omission.html (TL;DR: Use -target and -p measurementtype=hdrhistogram. Optionally,-p hdrhistogram.fileoutput=true|false -p hdrhistogram.output.path=path and [HistogramLogProcessor|https://github.com/HdrHistogram/HdrHistogram/blob/master/HistogramLogProcessor] and [plotFiles|https://github.com/HdrHistogram/HdrHistogram/blob/master/GoogleChartsExample/plotFiles.html] to plot latency by percentile.) I've been meaning to try it out. Hot contention spots; writing - Key: HBASE-12116 URL: https://issues.apache.org/jira/browse/HBASE-12116 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Attachments: 12116.checkForReplicas.txt, 12116.stringify.and.cache.scanner.maxsize.txt, 12116.txt, Screen Shot 2014-09-29 at 5.12.51 PM.png, Screen Shot 2014-09-30 at 10.39.34 PM.png, Screen Shot 2015-04-13 at 2.03.05 PM.png, perf.write3.svg, perf.write4.svg Playing with flight recorder, here are some write-time contentious synchronizations/locks (picture coming) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13307) Making methods under ScannerV2#next smaller allows them to be inlined gaining us 7% more throughput
[ https://issues.apache.org/jira/browse/HBASE-13307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-13307: -- Attachment: 13307v5.branch-1.txt Dirty patch. Making methods under ScannerV2#next smaller allows them to be inlined gaining us 7% more throughput --- Key: HBASE-13307 URL: https://issues.apache.org/jira/browse/HBASE-13307 Project: HBase Issue Type: Sub-task Components: Scanners Reporter: stack Assignee: stack Fix For: 2.0.0, 1.1.0 Attachments: 13307.txt, 13307v2.branch-1.txt, 13307v3.branch-1.txt, 13307v4.branch-1.txt, 13307v5.branch-1.txt, traces.8.svg, traces.9.svg See parent issue for patch and evidence. I was looking at graphs of our scan and found that methods were 'too big' to be inlined (looking at jvm compilation and inlining output flags -- see parent for list). Changing method size helped some. Let me commit the resultant patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12751) Allow RowLock to be reader writer
[ https://issues.apache.org/jira/browse/HBASE-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-12751: -- Attachment: HBASE-12751-v5.patch Letting HadoopQA chew on this one. This is the final merge of mvcc and sequenceid. Previously they were disjoint sets 1billion apart. However now it's a late binding mvcc and sequence id. The patch is a bit rough and the work done is a bit ugly. Pushing mvcc into WalKey is ugly as sin. However it's the best way I could think of to transit the mvcc down into fshlog. Allow RowLock to be reader writer - Key: HBASE-12751 URL: https://issues.apache.org/jira/browse/HBASE-12751 Project: HBase Issue Type: Bug Components: regionserver Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-12751-v1.patch, HBASE-12751-v2.patch, HBASE-12751-v3.patch, HBASE-12751-v4.patch, HBASE-12751-v5.patch, HBASE-12751.patch Right now every write operation grabs a row lock. This is to prevent values from changing during a read modify write operation (increment or check and put). However it limits parallelism in several different scenarios. If there are several puts to the same row but different columns or stores then this is very limiting. If there are puts to the same column then mvcc number should ensure a consistent ordering. So locking is not needed. However locking for check and put or increment is still needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12116) Hot contention spots; writing
[ https://issues.apache.org/jira/browse/HBASE-12116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494360#comment-14494360 ] Esteban Gutierrez commented on HBASE-12116: --- Good stuff [~apurtell]!. I'll try this out today. Hot contention spots; writing - Key: HBASE-12116 URL: https://issues.apache.org/jira/browse/HBASE-12116 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Attachments: 12116.checkForReplicas.txt, 12116.stringify.and.cache.scanner.maxsize.txt, 12116.txt, Screen Shot 2014-09-29 at 5.12.51 PM.png, Screen Shot 2014-09-30 at 10.39.34 PM.png, Screen Shot 2015-04-13 at 2.03.05 PM.png, perf.write3.svg, perf.write4.svg Playing with flight recorder, here are some write-time contentious synchronizations/locks (picture coming) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12751) Allow RowLock to be reader writer
[
https://issues.apache.org/jira/browse/HBASE-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494440#comment-14494440
]
Hadoop QA commented on HBASE-12751:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725267/HBASE-12751-v5.patch
against master branch at commit 4f151444b58ae85b93f76254961358932e0ffb9b.
ATTACHMENT ID: 12725267
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 94 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1928 checkstyle errors (more than the master's current 1910 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+this.htableDescriptor.getTableName(),
WALKey.NO_SEQUENCE_ID, nonceGroup, nonce, mvcc);
+ final WALEdit edits, final AtomicLong sequenceId, final boolean
inMemstore) throws IOException {
+ long logSeqNum, final long now, ListUUID clusterIds, long nonceGroup,
long nonce, MultiVersionConsistencyControl mvcc) {
+ final long now, ListUUID clusterIds, long nonceGroup, long nonce,
final MultiVersionConsistencyControl mvcc) {
+ final long now, ListUUID clusterIds, long nonceGroup, long nonce,
MultiVersionConsistencyControl mvcc) {
+ long logSeqNum, final long now, ListUUID clusterIds, long nonceGroup,
long nonce, MultiVersionConsistencyControl mvcc) {
+ long logSeqNum, final long now, ListUUID clusterIds, long nonceGroup,
long nonce, MultiVersionConsistencyControl mvcc) {
+ long logSeqNum, final long now, ListUUID clusterIds, long nonceGroup,
long nonce, MultiVersionConsistencyControl mvcc) {
+long txid = log.append(htd, info, getWalKey(System.currentTimeMillis()),
edit, sequenceId, true);
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestScanner
org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster
org.apache.hadoop.hbase.regionserver.wal.TestWALActionsListener
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13700//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13700//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13700//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13700//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13700//console
This message is automatically generated.
Allow RowLock to be reader writer
-
Key: HBASE-12751
URL: https://issues.apache.org/jira/browse/HBASE-12751
Project: HBase
Issue Type: Bug
Components: regionserver
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HBASE-12751-v1.patch, HBASE-12751-v2.patch,
HBASE-12751-v3.patch, HBASE-12751-v4.patch, HBASE-12751-v5.patch,
HBASE-12751.patch
Right now every write operation grabs a row lock. This is to prevent values
from changing during a read modify write operation (increment or check and
put). However it limits parallelism in several different scenarios.
If there are several puts to the same row but different columns or stores
then this is very limiting.
If there are puts to the same column then mvcc number should ensure a
consistent ordering. So locking is not needed.
However locking for check and put or increment is still needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12790) Support fairness across parallelized scans
[ https://issues.apache.org/jira/browse/HBASE-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494501#comment-14494501 ] ramkrishna.s.vasudevan commented on HBASE-12790: Am not very clear on how to write an end to end test case with actual scans. Anyway am trying to do some testing in the cluster. Will report back here. Support fairness across parallelized scans -- Key: HBASE-12790 URL: https://issues.apache.org/jira/browse/HBASE-12790 Project: HBase Issue Type: New Feature Reporter: James Taylor Assignee: ramkrishna.s.vasudevan Labels: Phoenix Attachments: AbstractRoundRobinQueue.java, HBASE-12790.patch, HBASE-12790_1.patch Some HBase clients parallelize the execution of a scan to reduce latency in getting back results. This can lead to starvation with a loaded cluster and interleaved scans, since the RPC queue will be ordered and processed on a FIFO basis. For example, if there are two clients, A B that submit largish scans at the same time. Say each scan is broken down into 100 scans by the client (broken down into equal depth chunks along the row key), and the 100 scans of client A are queued first, followed immediately by the 100 scans of client B. In this case, client B will be starved out of getting any results back until the scans for client A complete. One solution to this is to use the attached AbstractRoundRobinQueue instead of the standard FIFO queue. The queue to be used could be (maybe it already is) configurable based on a new config parameter. Using this queue would require the client to have the same identifier for all of the 100 parallel scans that represent a single logical scan from the clients point of view. With this information, the round robin queue would pick off a task from the queue in a round robin fashion (instead of a strictly FIFO manner) to prevent starvation over interleaved parallelized scans. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13460) Revise the MetaLookupPool executor-related defaults (introduced in HBASE-13036)
[
https://issues.apache.org/jira/browse/HBASE-13460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494553#comment-14494553
]
Enis Soztutar commented on HBASE-13460:
---
+1.
Revise the MetaLookupPool executor-related defaults (introduced in
HBASE-13036)
---
Key: HBASE-13460
URL: https://issues.apache.org/jira/browse/HBASE-13460
Project: HBase
Issue Type: Bug
Reporter: Devaraj Das
Assignee: Devaraj Das
Fix For: 1.1.0
Attachments: 13460-master-1.txt, 13460-master-2.txt,
13460-master-3.txt
The value of the max threads, max tasks, etc. can be improved to not
unnecessarily reject tasks. In one of our internal tests, we saw tasks
getting rejected (with the defaults that is currently there in the unpatched
code):
{noformat}
2015-04-09
12:36:34,111|beaver.machine|INFO|15134|140028818736896|MainThread|Caused by:
java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException:
Task
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@6658c97b
rejected from
java.util.concurrent.ThreadPoolExecutor@5a06[Running, pool size = 3,
active threads = 3, queued tasks = 300, completed tasks = 0]
2015-04-09
12:36:34,111|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208)
2015-04-09
12:36:34,112|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:145)
2015-04-09
12:36:34,112|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1246)
2015-04-09
12:36:34,112|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1152)
2015-04-09
12:36:34,113|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:370)
2015-04-09
12:36:34,113|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:321)
2015-04-09
12:36:34,113|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206)
2015-04-09
12:36:34,113|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
2015-04-09
12:36:34,114|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1414)
2015-04-09
12:36:34,114|beaver.machine|INFO|15134|140028818736896|MainThread|at
org.apache.hadoop.hbase.client.HTable.put(HTable.java:1010)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12006) [JDK 8] KeyStoreTestUtil#generateCertificate fails due to subject class type invalid
[
https://issues.apache.org/jira/browse/HBASE-12006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494520#comment-14494520
]
Srikanth Srungarapu commented on HBASE-12006:
-
Cross checked with the usages of {{X509V1CertificateGenerator}} available on
net. Tests seem to be passing against JDK 7 too.
+1 lgtm.
[JDK 8] KeyStoreTestUtil#generateCertificate fails due to subject class type
invalid
--
Key: HBASE-12006
URL: https://issues.apache.org/jira/browse/HBASE-12006
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0, 2.0.0
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
Fix For: 2.0.0, 1.1.0, 0.98.13, 1.0.2
Attachments: HBASE-12006.patch
Running tests on Java 8. All unit tests for branch 0.98 pass. On master
branch some variation in the security API is causing a failure in
TestSSLHttpServer:
{noformat}
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.181 sec
FAILURE! - in org.apache.hadoop.hbase.http.TestSSLHttpServer
org.apache.hadoop.hbase.http.TestSSLHttpServer Time elapsed: 0.181 sec
ERROR!
java.security.cert.CertificateException: Subject class type invalid.
at sun.security.x509.X509CertInfo.setSubject(X509CertInfo.java:888)
at sun.security.x509.X509CertInfo.set(X509CertInfo.java:415)
at
org.apache.hadoop.hbase.http.ssl.KeyStoreTestUtil.generateCertificate(KeyStoreTestUtil.java:94)
at
org.apache.hadoop.hbase.http.ssl.KeyStoreTestUtil.setupSSLConfig(KeyStoreTestUtil.java:246)
at
org.apache.hadoop.hbase.http.TestSSLHttpServer.setup(TestSSLHttpServer.java:72)
org.apache.hadoop.hbase.http.TestSSLHttpServer Time elapsed: 0.181 sec
ERROR!
java.lang.NullPointerException: null
at
org.apache.hadoop.hbase.http.TestSSLHttpServer.cleanup(TestSSLHttpServer.java:100)
Tests in error:
TestSSLHttpServer.setup:72 ยป Certificate Subject class type invalid.
TestSSLHttpServer.cleanup:100 NullPointer
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13453) Master should not bind to region server ports
[ https://issues.apache.org/jira/browse/HBASE-13453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srikanth Srungarapu updated HBASE-13453: Attachment: HBASE-13453.patch Master should not bind to region server ports - Key: HBASE-13453 URL: https://issues.apache.org/jira/browse/HBASE-13453 Project: HBase Issue Type: Improvement Reporter: Enis Soztutar Assignee: Devaraj Das Priority: Critical Fix For: 2.0.0, 1.1.0 Attachments: 34111-2.txt, HBASE-13453.patch, HBASE-13453.patch In 1.0, master by default binds to the region server ports (rpc and info). We have done it so thinking that in the long term, master and meta co-location will be default, and we can merge the master and region server as a single daemon. Over at HBASE-11165, if the conclusion end up being that meta will not be colocated at all, then master hosting a region server will just become an implementation detail. [~saint@gmail.com] says that we might never allow master to host regions. Now, we are stuck in a state where we have made master bind to RS ports in 1.0, which might create some confusion (and frustration) for small cluster users who traditionally used to host a master and a region server on the same node. I think we should undo this in 1.1 and use the previous master ports (16000) and not bind to 16030, so that the user does not need to do anything to bring up a RS on the same host. At least users going from 0.98 - 1.1 will not take a hit. Users going from 1.0 - 1.1 will see changed default ports. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13453) Master should not bind to region server ports
[
https://issues.apache.org/jira/browse/HBASE-13453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494525#comment-14494525
]
Srikanth Srungarapu commented on HBASE-13453:
-
bq. However, I would still like to see hbase-default to contain
{{hbase.master.port}} property.
Added the property in new patch.
Master should not bind to region server ports
-
Key: HBASE-13453
URL: https://issues.apache.org/jira/browse/HBASE-13453
Project: HBase
Issue Type: Improvement
Reporter: Enis Soztutar
Assignee: Devaraj Das
Priority: Critical
Fix For: 2.0.0, 1.1.0
Attachments: 34111-2.txt, HBASE-13453.patch, HBASE-13453.patch
In 1.0, master by default binds to the region server ports (rpc and info). We
have done it so thinking that in the long term, master and meta co-location
will be default, and we can merge the master and region server as a single
daemon.
Over at HBASE-11165, if the conclusion end up being that meta will not be
colocated at all, then master hosting a region server will just become an
implementation detail. [~saint@gmail.com] says that we might never allow
master to host regions.
Now, we are stuck in a state where we have made master bind to RS ports in
1.0, which might create some confusion (and frustration) for small cluster
users who traditionally used to host a master and a region server on the same
node.
I think we should undo this in 1.1 and use the previous master ports (16000)
and not bind to 16030, so that the user does not need to do anything to bring
up a RS on the same host. At least users going from 0.98 - 1.1 will not take
a hit. Users going from 1.0 - 1.1 will see changed default ports.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13466) Document deprecations in 1.x - Part 1
Lars Francke created HBASE-13466: Summary: Document deprecations in 1.x - Part 1 Key: HBASE-13466 URL: https://issues.apache.org/jira/browse/HBASE-13466 Project: HBase Issue Type: Task Reporter: Lars Francke Assignee: Lars Francke -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13203) Procedure v2 - master create/delete table
[ https://issues.apache.org/jira/browse/HBASE-13203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-13203: Attachment: HBASE-13203-v1-branch-1.patch Procedure v2 - master create/delete table - Key: HBASE-13203 URL: https://issues.apache.org/jira/browse/HBASE-13203 Project: HBase Issue Type: Sub-task Components: master Affects Versions: 2.0.0, 1.1.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 2.0.0 Attachments: HBASE-13203-v0.patch, HBASE-13203-v1-branch-1.patch, HBASE-13203-v1.patch master side, part of HBASE-12439 starts up the procedure executor on the master and replaces the create/delete table handlers with the procedure version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13211) Procedure V2 - master Enable/Disable table
[ https://issues.apache.org/jira/browse/HBASE-13211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-13211: Attachment: HBASE-13211-v2-branch-1.patch Procedure V2 - master Enable/Disable table -- Key: HBASE-13211 URL: https://issues.apache.org/jira/browse/HBASE-13211 Project: HBase Issue Type: Sub-task Components: master Affects Versions: 2.0.0, 1.1.0 Reporter: Stephen Yuan Jiang Assignee: Stephen Yuan Jiang Labels: reliability Fix For: 2.0.0 Attachments: EnableDisableTableServer-no-gen-files.v1-master.patch, HBASE-13211-v2-branch-1.patch, HBASE-13211-v2.patch Original Estimate: 120h Time Spent: 216h Remaining Estimate: 0h master side, part of HBASE-12439 starts up the procedure executor on the master and replaces the enable/disable table handlers with the procedure version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13307) Making methods under ScannerV2#next smaller allows them to be inlined gaining us 7% more throughput
[
https://issues.apache.org/jira/browse/HBASE-13307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494257#comment-14494257
]
Hadoop QA commented on HBASE-13307:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725250/13307v5.branch-1.txt
against branch-1 branch at commit 4f151444b58ae85b93f76254961358932e0ffb9b.
ATTACHMENT ID: 12725250
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13699//console
This message is automatically generated.
Making methods under ScannerV2#next smaller allows them to be inlined gaining
us 7% more throughput
---
Key: HBASE-13307
URL: https://issues.apache.org/jira/browse/HBASE-13307
Project: HBase
Issue Type: Sub-task
Components: Scanners
Reporter: stack
Assignee: stack
Fix For: 2.0.0, 1.1.0
Attachments: 13307.txt, 13307v2.branch-1.txt, 13307v3.branch-1.txt,
13307v4.branch-1.txt, 13307v5.branch-1.txt, traces.8.svg, traces.9.svg
See parent issue for patch and evidence.
I was looking at graphs of our scan and found that methods were 'too big' to
be inlined (looking at jvm compilation and inlining output flags -- see
parent for list). Changing method size helped some. Let me commit the
resultant patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13466) Document deprecations in 1.x - Part 1
[ https://issues.apache.org/jira/browse/HBASE-13466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HBASE-13466: - Status: Patch Available (was: Open) Document deprecations in 1.x - Part 1 - Key: HBASE-13466 URL: https://issues.apache.org/jira/browse/HBASE-13466 Project: HBase Issue Type: Sub-task Reporter: Lars Francke Assignee: Lars Francke Attachments: HBASE-13466.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13466) Document deprecations in 1.x - Part 1
[ https://issues.apache.org/jira/browse/HBASE-13466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HBASE-13466: - Attachment: HBASE-13466.1.patch Document deprecations in 1.x - Part 1 - Key: HBASE-13466 URL: https://issues.apache.org/jira/browse/HBASE-13466 Project: HBase Issue Type: Sub-task Reporter: Lars Francke Assignee: Lars Francke Attachments: HBASE-13466.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13290) Procedure v2 - client enable/disable table sync
[ https://issues.apache.org/jira/browse/HBASE-13290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-13290: Attachment: HBASE-13290-v2-branch-1.patch Procedure v2 - client enable/disable table sync --- Key: HBASE-13290 URL: https://issues.apache.org/jira/browse/HBASE-13290 Project: HBase Issue Type: Sub-task Components: Client Affects Versions: 2.0.0, 1.1.0 Reporter: Stephen Yuan Jiang Assignee: Stephen Yuan Jiang Fix For: 2.0.0 Attachments: EnableDisableTableClientPV2-draft.v0-master.patch, HBASE-13290-v2-branch-1.patch, HBASE-13290-v2.patch Original Estimate: 120h Time Spent: 48h Remaining Estimate: 72h client side, part of HBASE-12439/HBASE-13211 it uses the new procedure code to be know when the procedure is completed, and have a proper sync/async behavior on enable/disable table. For 1.1, It has to be binary compatible (1.0 client can talk to 1.1 server 1.1 client can talk to 1.0 server). Binary compatible is TBD for 2.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12006) [JDK 8] KeyStoreTestUtil#generateCertificate fails due to subject class type invalid
[
https://issues.apache.org/jira/browse/HBASE-12006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494164#comment-14494164
]
Lars Francke commented on HBASE-12006:
--
Thanks [~apurtell]. I've tried this patch and it fixes the bug for me as well.
Would love to see this committed.
[JDK 8] KeyStoreTestUtil#generateCertificate fails due to subject class type
invalid
--
Key: HBASE-12006
URL: https://issues.apache.org/jira/browse/HBASE-12006
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0, 2.0.0
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
Fix For: 2.0.0, 1.1.0, 0.98.13, 1.0.2
Attachments: HBASE-12006.patch
Running tests on Java 8. All unit tests for branch 0.98 pass. On master
branch some variation in the security API is causing a failure in
TestSSLHttpServer:
{noformat}
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.181 sec
FAILURE! - in org.apache.hadoop.hbase.http.TestSSLHttpServer
org.apache.hadoop.hbase.http.TestSSLHttpServer Time elapsed: 0.181 sec
ERROR!
java.security.cert.CertificateException: Subject class type invalid.
at sun.security.x509.X509CertInfo.setSubject(X509CertInfo.java:888)
at sun.security.x509.X509CertInfo.set(X509CertInfo.java:415)
at
org.apache.hadoop.hbase.http.ssl.KeyStoreTestUtil.generateCertificate(KeyStoreTestUtil.java:94)
at
org.apache.hadoop.hbase.http.ssl.KeyStoreTestUtil.setupSSLConfig(KeyStoreTestUtil.java:246)
at
org.apache.hadoop.hbase.http.TestSSLHttpServer.setup(TestSSLHttpServer.java:72)
org.apache.hadoop.hbase.http.TestSSLHttpServer Time elapsed: 0.181 sec
ERROR!
java.lang.NullPointerException: null
at
org.apache.hadoop.hbase.http.TestSSLHttpServer.cleanup(TestSSLHttpServer.java:100)
Tests in error:
TestSSLHttpServer.setup:72 ยป Certificate Subject class type invalid.
TestSSLHttpServer.cleanup:100 NullPointer
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13465) Document deprecations in 1.x
Lars Francke created HBASE-13465: Summary: Document deprecations in 1.x Key: HBASE-13465 URL: https://issues.apache.org/jira/browse/HBASE-13465 Project: HBase Issue Type: Umbrella Reporter: Lars Francke -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-13464) Remove deprecations for 2.0.0 - Part 1
[
https://issues.apache.org/jira/browse/HBASE-13464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Francke reassigned HBASE-13464:
Assignee: Lars Francke
Remove deprecations for 2.0.0 - Part 1
--
Key: HBASE-13464
URL: https://issues.apache.org/jira/browse/HBASE-13464
Project: HBase
Issue Type: Sub-task
Reporter: Lars Francke
Assignee: Lars Francke
This removes deprecations for the following issues:
* HBASE-6038
* HBASE-1502
* HBASE-5453
* HBASE-5357
* HBASE-9870
* HBASE-10870
* HBASE-12363
* HBASE-9508
It introduces one new deprecation for
{{UnmodifyableHTableDescriptor#UnmodifyableHTableDescriptor()}} because that
depends on a deprecated constructor in {{HTableDescriptor}}. I've made the
constructor in question {{protected}} so both can be removed in 3.0.0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13464) Remove deprecations for 2.0.0 - Part 1
Lars Francke created HBASE-13464:
Summary: Remove deprecations for 2.0.0 - Part 1
Key: HBASE-13464
URL: https://issues.apache.org/jira/browse/HBASE-13464
Project: HBase
Issue Type: Sub-task
Reporter: Lars Francke
This removes deprecations for the following issues:
* HBASE-6038
* HBASE-1502
* HBASE-5453
* HBASE-5357
* HBASE-9870
* HBASE-10870
* HBASE-12363
* HBASE-9508
It introduces one new deprecation for
{{UnmodifyableHTableDescriptor#UnmodifyableHTableDescriptor()}} because that
depends on a deprecated constructor in {{HTableDescriptor}}. I've made the
constructor in question {{protected}} so both can be removed in 3.0.0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13465) Document deprecations in 1.x
[ https://issues.apache.org/jira/browse/HBASE-13465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HBASE-13465: - Description: This is an umbrella issue for everything that deals with deprecated code for the HBase 1.x branch. This was discussed on the [mailing list|http://apache-hbase.679495.n3.nabble.com/Removal-of-deprecated-features-td4069746.html] and agreed upon: * Document in which version a deprecated feature will be removed, when it was initially removed and in which JIRA issue Document deprecations in 1.x Key: HBASE-13465 URL: https://issues.apache.org/jira/browse/HBASE-13465 Project: HBase Issue Type: Umbrella Reporter: Lars Francke This is an umbrella issue for everything that deals with deprecated code for the HBase 1.x branch. This was discussed on the [mailing list|http://apache-hbase.679495.n3.nabble.com/Removal-of-deprecated-features-td4069746.html] and agreed upon: * Document in which version a deprecated feature will be removed, when it was initially removed and in which JIRA issue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13466) Document deprecations in 1.x - Part 1
[
https://issues.apache.org/jira/browse/HBASE-13466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494250#comment-14494250
]
Hadoop QA commented on HBASE-13466:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725248/HBASE-13466.1.patch
against master branch at commit 4f151444b58ae85b93f76254961358932e0ffb9b.
ATTACHMENT ID: 12725248
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13698//console
This message is automatically generated.
Document deprecations in 1.x - Part 1
-
Key: HBASE-13466
URL: https://issues.apache.org/jira/browse/HBASE-13466
Project: HBase
Issue Type: Sub-task
Reporter: Lars Francke
Assignee: Lars Francke
Attachments: HBASE-13466.1.patch
This documents deprecations for the following issues:
* HBASE-6038
* HBASE-1502
* HBASE-5453
* HBASE-5357
* HBASE-9870
* HBASE-10870
* HBASE-12363
* HBASE-9508
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12006) [JDK 8] KeyStoreTestUtil#generateCertificate fails due to subject class type invalid
[
https://issues.apache.org/jira/browse/HBASE-12006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494293#comment-14494293
]
Andrew Purtell commented on HBASE-12006:
Can I get a quick review?
Mind if I bug you for a sec [~stack] or [~srikanth235] or [~jerryhe] ?
[JDK 8] KeyStoreTestUtil#generateCertificate fails due to subject class type
invalid
--
Key: HBASE-12006
URL: https://issues.apache.org/jira/browse/HBASE-12006
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0, 2.0.0
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
Fix For: 2.0.0, 1.1.0, 0.98.13, 1.0.2
Attachments: HBASE-12006.patch
Running tests on Java 8. All unit tests for branch 0.98 pass. On master
branch some variation in the security API is causing a failure in
TestSSLHttpServer:
{noformat}
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.181 sec
FAILURE! - in org.apache.hadoop.hbase.http.TestSSLHttpServer
org.apache.hadoop.hbase.http.TestSSLHttpServer Time elapsed: 0.181 sec
ERROR!
java.security.cert.CertificateException: Subject class type invalid.
at sun.security.x509.X509CertInfo.setSubject(X509CertInfo.java:888)
at sun.security.x509.X509CertInfo.set(X509CertInfo.java:415)
at
org.apache.hadoop.hbase.http.ssl.KeyStoreTestUtil.generateCertificate(KeyStoreTestUtil.java:94)
at
org.apache.hadoop.hbase.http.ssl.KeyStoreTestUtil.setupSSLConfig(KeyStoreTestUtil.java:246)
at
org.apache.hadoop.hbase.http.TestSSLHttpServer.setup(TestSSLHttpServer.java:72)
org.apache.hadoop.hbase.http.TestSSLHttpServer Time elapsed: 0.181 sec
ERROR!
java.lang.NullPointerException: null
at
org.apache.hadoop.hbase.http.TestSSLHttpServer.cleanup(TestSSLHttpServer.java:100)
Tests in error:
TestSSLHttpServer.setup:72 ยป Certificate Subject class type invalid.
TestSSLHttpServer.cleanup:100 NullPointer
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13463) Quota needs to be updated only in case of successful region merge
[
https://issues.apache.org/jira/browse/HBASE-13463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494316#comment-14494316
]
Hadoop QA commented on HBASE-13463:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725216/13463-v2-branch-1.txt
against branch-1 branch at commit 4f151444b58ae85b93f76254961358932e0ffb9b.
ATTACHMENT ID: 12725216
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13696//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13696//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13696//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13696//console
This message is automatically generated.
Quota needs to be updated only in case of successful region merge
-
Key: HBASE-13463
URL: https://issues.apache.org/jira/browse/HBASE-13463
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 1.1.0
Attachments: 13463-v1.txt, 13463-v2-branch-1.txt
During review of HBASE-13438, Vandana pointed out the following:
in the case of MERGE_REVERTED, the assignment manager is not updating the
quota. You need to add another hook in RegionStateListener class just like
split.
This issue is to fix the above case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13464) Remove deprecations for 2.0.0 - Part 1
[
https://issues.apache.org/jira/browse/HBASE-13464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Francke updated HBASE-13464:
-
Attachment: HBASE-13464.1.patch
Remove deprecations for 2.0.0 - Part 1
--
Key: HBASE-13464
URL: https://issues.apache.org/jira/browse/HBASE-13464
Project: HBase
Issue Type: Sub-task
Reporter: Lars Francke
Assignee: Lars Francke
Attachments: HBASE-13464.1.patch
This removes deprecations for the following issues:
* HBASE-6038
* HBASE-1502
* HBASE-5453
* HBASE-5357
* HBASE-9870
* HBASE-10870
* HBASE-12363
* HBASE-9508
It introduces one new deprecation for
{{UnmodifyableHTableDescriptor#UnmodifyableHTableDescriptor()}} because that
depends on a deprecated constructor in {{HTableDescriptor}}. I've made the
constructor in question {{protected}} so both can be removed in 3.0.0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13464) Remove deprecations for 2.0.0 - Part 1
[
https://issues.apache.org/jira/browse/HBASE-13464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Francke updated HBASE-13464:
-
Status: Patch Available (was: Open)
Remove deprecations for 2.0.0 - Part 1
--
Key: HBASE-13464
URL: https://issues.apache.org/jira/browse/HBASE-13464
Project: HBase
Issue Type: Sub-task
Reporter: Lars Francke
Assignee: Lars Francke
Attachments: HBASE-13464.1.patch
This removes deprecations for the following issues:
* HBASE-6038
* HBASE-1502
* HBASE-5453
* HBASE-5357
* HBASE-9870
* HBASE-10870
* HBASE-12363
* HBASE-9508
It introduces one new deprecation for
{{UnmodifyableHTableDescriptor#UnmodifyableHTableDescriptor()}} because that
depends on a deprecated constructor in {{HTableDescriptor}}. I've made the
constructor in question {{protected}} so both can be removed in 3.0.0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13301) Possible memory leak in BucketCache
[
https://issues.apache.org/jira/browse/HBASE-13301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494236#comment-14494236
]
Hudson commented on HBASE-13301:
SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #901 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/901/])
HBASE-13301 Possible memory leak in BucketCache (zhangduo: rev
0169cbfde8967044ef1248e8fa91dcd8abc83f14)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/CachedEntryQueue.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketCache.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/IdLock.java
Possible memory leak in BucketCache
---
Key: HBASE-13301
URL: https://issues.apache.org/jira/browse/HBASE-13301
Project: HBase
Issue Type: Bug
Components: BlockCache
Reporter: zhangduo
Assignee: zhangduo
Fix For: 2.0.0, 1.1.0, 0.98.13, 1.0.2
Attachments: HBASE-13301-0.98.patch, HBASE-13301-0.98_v1.patch,
HBASE-13301-branch-1.0.patch, HBASE-13301-branch-1.0.patch,
HBASE-13301-branch-1.0_v1.patch, HBASE-13301-branch-1.patch,
HBASE-13301-branch-1_v1.patch, HBASE-13301-testcase.patch,
HBASE-13301-testcase_v1.patch, HBASE-13301.patch, HBASE-13301_v1.patch,
HBASE-13301_v2.patch, HBASE-13301_v3.patch
{code:title=BucketCache.java}
public boolean evictBlock(BlockCacheKey cacheKey) {
...
if (bucketEntry.equals(backingMap.remove(cacheKey))) {
bucketAllocator.freeBlock(bucketEntry.offset());
realCacheSize.addAndGet(-1 * bucketEntry.getLength());
blocksByHFile.remove(cacheKey.getHfileName(), cacheKey);
if (removedBlock == null) {
this.blockNumber.decrementAndGet();
}
} else {
return false;
}
...
{code}
I think the problem is here. We remove a BucketEntry that should not be
removed by us, but we do not put it back and also do not do any clean up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12116) Hot contention spots; writing
[ https://issues.apache.org/jira/browse/HBASE-12116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494176#comment-14494176 ] Andrew Purtell commented on HBASE-12116: bq. CSLM#doPut at 12% Revive the CSLM alternative JIRAs? (HBASE-3484, HBASE-3993 (+subtasks), HBASE-10713) Hot contention spots; writing - Key: HBASE-12116 URL: https://issues.apache.org/jira/browse/HBASE-12116 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Attachments: 12116.checkForReplicas.txt, 12116.stringify.and.cache.scanner.maxsize.txt, 12116.txt, Screen Shot 2014-09-29 at 5.12.51 PM.png, Screen Shot 2014-09-30 at 10.39.34 PM.png, Screen Shot 2015-04-13 at 2.03.05 PM.png, perf.write3.svg, perf.write4.svg Playing with flight recorder, here are some write-time contentious synchronizations/locks (picture coming) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13466) Document deprecations in 1.x - Part 1
[ https://issues.apache.org/jira/browse/HBASE-13466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HBASE-13466: - Issue Type: Sub-task (was: Task) Parent: HBASE-13465 Document deprecations in 1.x - Part 1 - Key: HBASE-13466 URL: https://issues.apache.org/jira/browse/HBASE-13466 Project: HBase Issue Type: Sub-task Reporter: Lars Francke Assignee: Lars Francke -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13209) Procedure V2 - master Add/Modify/Delete Column Family
[ https://issues.apache.org/jira/browse/HBASE-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-13209: Attachment: HBASE-13209-v3-branch-1.patch Procedure V2 - master Add/Modify/Delete Column Family - Key: HBASE-13209 URL: https://issues.apache.org/jira/browse/HBASE-13209 Project: HBase Issue Type: Sub-task Components: master Affects Versions: 2.0.0, 1.1.0 Reporter: Stephen Yuan Jiang Assignee: Stephen Yuan Jiang Labels: reliability Fix For: 2.0.0 Attachments: AlterColumnFamily-no-gen-file.v1-master.patch, HBASE-13209-v2.patch, HBASE-13209-v3-branch-1.patch, HBASE-13209-v3.patch Original Estimate: 168h Time Spent: 384h Remaining Estimate: 0h master side, part of HBASE-12439 starts up the procedure executor on the master and replaces the add/modify/delete handlers with the procedure version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13375) Provide HBase superuser higher priority over other users in the RPC handling
[ https://issues.apache.org/jira/browse/HBASE-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495433#comment-14495433 ] Devaraj Das commented on HBASE-13375: - [~mantonov] do we know why the test failed with fewer handlers? Provide HBase superuser higher priority over other users in the RPC handling Key: HBASE-13375 URL: https://issues.apache.org/jira/browse/HBASE-13375 Project: HBase Issue Type: Improvement Components: rpc Reporter: Devaraj Das Assignee: Mikhail Antonov Fix For: 1.1.0 Attachments: HBASE-13375-v0.patch, HBASE-13375-v1.patch, HBASE-13375-v1.patch, HBASE-13375-v1.patch, HBASE-13375-v2.patch HBASE-13351 annotates Master RPCs so that RegionServer RPCs are treated with a higher priority compared to user RPCs (and they are handled by a separate set of handlers, etc.). It may be good to stretch this to users too - hbase superuser (configured via hbase.superuser) gets higher priority over other users in the RPC handling. That way the superuser can always perform administrative operations on the cluster even if all the normal priority handlers are occupied (for example, we had a situation where all the master's handlers were tied up with many simultaneous createTable RPC calls from multiple users and the master wasn't able to perform any operations initiated by the admin). (Discussed this some with [~enis] and [~elserj]). Does this make sense to others? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13467) Prototype using GRPC as IPC mechanism
[ https://issues.apache.org/jira/browse/HBASE-13467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495447#comment-14495447 ] zhangduo commented on HBASE-13467: -- Nice try! Two things. 1. Wire compatibility. gRPC is based on HTTP/2, and the old rpc is based on raw TCP. If we can not keep compatibility at the protocol level, then we should find other ways to let people use old client communicate with new server. 2. Secure HBase. gRPC is based on HTTP/2, so I'm not worried about the kerberos authentication part. But security is a big system, a little change here may require large change there. It maybe a big project. Thanks. Prototype using GRPC as IPC mechanism - Key: HBASE-13467 URL: https://issues.apache.org/jira/browse/HBASE-13467 Project: HBase Issue Type: Improvement Components: API Affects Versions: 2.0.0 Reporter: Louis Ryan Priority: Minor GRPC provide an RPC layer for protocol buffers on top of Netty 4/5. This could be used to replace the current internal implementation. GRPC supports some advanced features like streaming, async, flow-control, cancellation timeout which might be useful Will prototype on GitHub here if folks are interested https://github.com/louiscryan/hbase -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13351) Annotate internal MasterRpcServices methods with admin priority
[
https://issues.apache.org/jira/browse/HBASE-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495460#comment-14495460
]
Josh Elser commented on HBASE-13351:
It should.
{panel:title=TableStateManager.java}
{code}
@Nullable
protected TableState readMetaState(TableName tableName) throws IOException {
if (tableName.equals(TableName.META_TABLE_NAME))
return new TableState(tableName, TableState.State.ENABLED);
return MetaTableAccessor.getTableState(master.getConnection(), tableName);
}
{code}
{panel}
When we read the state of a table from meta, we should be reusing the Master's
cached connection (which has the opportunity to give us the short-circuit
variant).
The question I have, is are we actually getting a ShortCircuit connection when
we think we should be
{panel:title=ConnectionUtils.java}
{code}
@Override
public ClientService.BlockingInterface getClient(
ServerName sn) throws IOException {
return serverName.equals(sn) ? client : super.getClient(sn);
}
{code}
{panel}
My current hunch is that we're failing that conditional for one reason or
another. Will dig into this next.
Annotate internal MasterRpcServices methods with admin priority
---
Key: HBASE-13351
URL: https://issues.apache.org/jira/browse/HBASE-13351
Project: HBase
Issue Type: Improvement
Components: master
Reporter: Josh Elser
Assignee: Josh Elser
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13351-v1.patch, HBASE-13351-v2.patch,
HBASE-13351-v3.patch, HBASE-13351.patch
HBASE-12071, among other things, introduced annotating RPC methods to give
certain methods priority over others. Namely, this helps ensure that client
requests cannot starve out internal RPC between master and regionserver.
Similarly, we can do the same thing for Master RPC methods that are invoked
by RS's.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13453) Master should not bind to region server ports
[ https://issues.apache.org/jira/browse/HBASE-13453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srikanth Srungarapu updated HBASE-13453: Status: Patch Available (was: Open) Master should not bind to region server ports - Key: HBASE-13453 URL: https://issues.apache.org/jira/browse/HBASE-13453 Project: HBase Issue Type: Improvement Reporter: Enis Soztutar Assignee: Devaraj Das Priority: Critical Fix For: 2.0.0, 1.1.0 Attachments: 34111-2.txt, HBASE-13453.patch, HBASE-13453.patch In 1.0, master by default binds to the region server ports (rpc and info). We have done it so thinking that in the long term, master and meta co-location will be default, and we can merge the master and region server as a single daemon. Over at HBASE-11165, if the conclusion end up being that meta will not be colocated at all, then master hosting a region server will just become an implementation detail. [~saint@gmail.com] says that we might never allow master to host regions. Now, we are stuck in a state where we have made master bind to RS ports in 1.0, which might create some confusion (and frustration) for small cluster users who traditionally used to host a master and a region server on the same node. I think we should undo this in 1.1 and use the previous master ports (16000) and not bind to 16030, so that the user does not need to do anything to bring up a RS on the same host. At least users going from 0.98 - 1.1 will not take a hit. Users going from 1.0 - 1.1 will see changed default ports. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13469) [branch-1] Procedure V2 - Make procedure v2 configurable in branch-1
[ https://issues.apache.org/jira/browse/HBASE-13469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495508#comment-14495508 ] Enis Soztutar commented on HBASE-13469: --- bq. I'd say -0. Cut over. Just kill the old stuff. Test. No safety net! Bold! What about keeping it configurable in 1.1, and removing the conf in 1.2+? [branch-1] Procedure V2 - Make procedure v2 configurable in branch-1 Key: HBASE-13469 URL: https://issues.apache.org/jira/browse/HBASE-13469 Project: HBase Issue Type: Sub-task Components: master Reporter: Enis Soztutar Fix For: 1.1.0 In branch-1, I think we want proc v2 to be configurable, so that if any non-recoverable issue is found, at least there is a workaround. We already have the handlers and code laying around. It will be just introducing the config to enable / disable. We can even make it dynamically configurable via the new framework. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13472) Polish IN_MEMORY table behavior
[ https://issues.apache.org/jira/browse/HBASE-13472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495545#comment-14495545 ] Andrew Purtell commented on HBASE-13472: On trunk we could go so far as to make IN_MEMORY a synonym for other attributes. Polish IN_MEMORY table behavior --- Key: HBASE-13472 URL: https://issues.apache.org/jira/browse/HBASE-13472 Project: HBase Issue Type: Task Reporter: Andrew Purtell For a long time we've been able to support a mode of operation that keeps as much table data as possible in memory, so HBase can be used as an 'in-memory' DB with fully durable WAL and write-behind persistence of table data. However: - There are a set of relevant schema options (IN_MEMORY, CACHE_ON_WRITE, PREFETCH_BLOCKS_ON_OPEN, block encoding), so set up isn't simple. We should have a shortcut that sets all this up in one place. I'm thinking a utility class with static helpers that configure a table descriptor with all of the needed bits. (Other ideas?) - We don't have a safety valve. An in-memory table can become too large, where it crowds out all blockcache space for other tables on the cluster, where it falls out of blockcache and performs poorly without warning because it's become too big. Consider table quota support with an option for region size limits as % of total heap consumed by regions for a given table. Warn at soft limit. Refuse writes if over hard limit. Follow on work can investigate options hooking up to offheap work. That's not in scope here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12987) HBCK should print status while scanning over many regions
[
https://issues.apache.org/jira/browse/HBASE-12987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495554#comment-14495554
]
Hadoop QA commented on HBASE-12987:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725427/HBASE-12987-branch-1-v1.patch
against branch-1 branch at commit 4f151444b58ae85b93f76254961358932e0ffb9b.
ATTACHMENT ID: 12725427
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13704//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13704//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13704//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13704//console
This message is automatically generated.
HBCK should print status while scanning over many regions
-
Key: HBASE-12987
URL: https://issues.apache.org/jira/browse/HBASE-12987
Project: HBase
Issue Type: Improvement
Components: hbck, Usability
Reporter: Nick Dimiduk
Assignee: Josh Elser
Labels: beginner
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-12987-branch-1-v1.patch,
HBASE-12987-branch-1.patch, HBASE-12987-master.patch, hbck-output.txt
Running simple commands like hbck -summary on a large table can take some
time. We should print some information to let it be known how things are
progressing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13453) Master should not bind to region server ports
[
https://issues.apache.org/jira/browse/HBASE-13453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495602#comment-14495602
]
Hadoop QA commented on HBASE-13453:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725300/HBASE-13453.patch
against master branch at commit 4f151444b58ae85b93f76254961358932e0ffb9b.
ATTACHMENT ID: 12725300
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13705//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13705//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13705//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13705//console
This message is automatically generated.
Master should not bind to region server ports
-
Key: HBASE-13453
URL: https://issues.apache.org/jira/browse/HBASE-13453
Project: HBase
Issue Type: Improvement
Reporter: Enis Soztutar
Assignee: Devaraj Das
Priority: Critical
Fix For: 2.0.0, 1.1.0
Attachments: 34111-2.txt, HBASE-13453.patch, HBASE-13453.patch
In 1.0, master by default binds to the region server ports (rpc and info). We
have done it so thinking that in the long term, master and meta co-location
will be default, and we can merge the master and region server as a single
daemon.
Over at HBASE-11165, if the conclusion end up being that meta will not be
colocated at all, then master hosting a region server will just become an
implementation detail. [~saint@gmail.com] says that we might never allow
master to host regions.
Now, we are stuck in a state where we have made master bind to RS ports in
1.0, which might create some confusion (and frustration) for small cluster
users who traditionally used to host a master and a region server on the same
node.
I think we should undo this in 1.1 and use the previous master ports (16000)
and not bind to 16030, so that the user does not need to do anything to bring
up a RS on the same host. At least users going from 0.98 - 1.1 will not take
a hit. Users going from 1.0 - 1.1 will see changed default ports.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13375) Provide HBase superuser higher priority over other users in the RPC handling
[ https://issues.apache.org/jira/browse/HBASE-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495441#comment-14495441 ] Josh Elser commented on HBASE-13375: [~devaraj] see https://issues.apache.org/jira/browse/HBASE-13351?focusedCommentId=14494848page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14494848 Provide HBase superuser higher priority over other users in the RPC handling Key: HBASE-13375 URL: https://issues.apache.org/jira/browse/HBASE-13375 Project: HBase Issue Type: Improvement Components: rpc Reporter: Devaraj Das Assignee: Mikhail Antonov Fix For: 1.1.0 Attachments: HBASE-13375-v0.patch, HBASE-13375-v1.patch, HBASE-13375-v1.patch, HBASE-13375-v1.patch, HBASE-13375-v2.patch HBASE-13351 annotates Master RPCs so that RegionServer RPCs are treated with a higher priority compared to user RPCs (and they are handled by a separate set of handlers, etc.). It may be good to stretch this to users too - hbase superuser (configured via hbase.superuser) gets higher priority over other users in the RPC handling. That way the superuser can always perform administrative operations on the cluster even if all the normal priority handlers are occupied (for example, we had a situation where all the master's handlers were tied up with many simultaneous createTable RPC calls from multiple users and the master wasn't able to perform any operations initiated by the admin). (Discussed this some with [~enis] and [~elserj]). Does this make sense to others? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13472) Polish IN_MEMORY table behavior
Andrew Purtell created HBASE-13472: -- Summary: Polish IN_MEMORY table behavior Key: HBASE-13472 URL: https://issues.apache.org/jira/browse/HBASE-13472 Project: HBase Issue Type: Task Reporter: Andrew Purtell For a long time we've been able to support a mode of operation that keeps as much table data as possible in memory, so HBase can be used as an 'in-memory' DB with fully durable WAL and write-behind persistence of table data. However: - There are a set of relevant schema options (IN_MEMORY, CACHE_ON_WRITE, PREFETCH_BLOCKS_ON_OPEN, block encoding), so set up isn't simple. We should have a shortcut that sets all this up in one place. I'm thinking a utility class with static helpers that configure a table descriptor with all of the needed bits. (Other ideas?) - We don't have a safety valve. An in-memory table can become too large, where it crowds out all blockcache space for other tables on the cluster, where it falls out of blockcache and performs poorly without warning because it's become too big. Consider table quota support with an option for region size limits as % of total heap consumed by regions for a given table. Warn at soft limit. Refuse writes if over hard limit. Follow on work can investigate options hooking up to offheap work. That's not in scope here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12987) HBCK should print status while scanning over many regions
[ https://issues.apache.org/jira/browse/HBASE-12987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-12987: --- Attachment: HBASE-12987-branch-1-v1.patch Still confused why the old patch failed to apply, but let's try this one. HBCK should print status while scanning over many regions - Key: HBASE-12987 URL: https://issues.apache.org/jira/browse/HBASE-12987 Project: HBase Issue Type: Improvement Components: hbck, Usability Reporter: Nick Dimiduk Assignee: Josh Elser Labels: beginner Fix For: 2.0.0, 1.1.0 Attachments: HBASE-12987-branch-1-v1.patch, HBASE-12987-branch-1.patch, HBASE-12987-master.patch, hbck-output.txt Running simple commands like hbck -summary on a large table can take some time. We should print some information to let it be known how things are progressing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13463) Quota needs to be updated only in case of successful region merge
[
https://issues.apache.org/jira/browse/HBASE-13463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495204#comment-14495204
]
Vandana Ayyalasomayajula commented on HBASE-13463:
--
[~tedyu] -- The idea of updating quota only on successful merge sounds good, as
I think concurrent merge operations would be rare unlike splits.
looking at the v2 patch,
{quote}
if (code == TransitionCode.MERGED
org.apache.commons.lang.StringUtils.isEmpty(errorMsg)) {
+try {
+ regionStateListener.onRegionMerged(hri);
+} catch (IOException exp) {
+ errorMsg = StringUtils.stringifyException(exp);
+}
+ }
{quote}
shouldn't that be under MERGED case rather than MERGE_REVERTED ?
Quota needs to be updated only in case of successful region merge
-
Key: HBASE-13463
URL: https://issues.apache.org/jira/browse/HBASE-13463
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 1.1.0
Attachments: 13463-v1.txt, 13463-v2-branch-1.txt
During review of HBASE-13438, Vandana pointed out the following:
in the case of MERGE_REVERTED, the assignment manager is not updating the
quota. You need to add another hook in RegionStateListener class just like
split.
This issue is to fix the above case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13463) Quota needs to be updated only in case of successful region merge
[ https://issues.apache.org/jira/browse/HBASE-13463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495505#comment-14495505 ] Ted Yu commented on HBASE-13463: The intention is to put call to listener after the call to onRegionMerge(). This aligns with the formation in master branch. Quota needs to be updated only in case of successful region merge - Key: HBASE-13463 URL: https://issues.apache.org/jira/browse/HBASE-13463 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Fix For: 1.1.0 Attachments: 13463-v1.txt, 13463-v2-branch-1.txt During review of HBASE-13438, Vandana pointed out the following: in the case of MERGE_REVERTED, the assignment manager is not updating the quota. You need to add another hook in RegionStateListener class just like split. This issue is to fix the above case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13471) CompactionRequest can deadlock closing a region
[
https://issues.apache.org/jira/browse/HBASE-13471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HBASE-13471:
--
Assignee: Rajesh Nishtala (was: Elliott Clark)
CompactionRequest can deadlock closing a region
---
Key: HBASE-13471
URL: https://issues.apache.org/jira/browse/HBASE-13471
Project: HBase
Issue Type: Bug
Reporter: Elliott Clark
Assignee: Rajesh Nishtala
{code}
Thread 4139
(regionserver/hbase412.example.com/10.158.6.53:60020-splits-1429003183537):
State: WAITING
Blocked count: 131
Waited count: 228
Waiting on
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@50714dc3
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1371)
org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1325)
org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:352)
org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:252)
org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:509)
org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:84)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13471) Deadlock closing a region
[
https://issues.apache.org/jira/browse/HBASE-13471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HBASE-13471:
--
Summary: Deadlock closing a region (was: CompactionRequest can deadlock
closing a region)
Deadlock closing a region
-
Key: HBASE-13471
URL: https://issues.apache.org/jira/browse/HBASE-13471
Project: HBase
Issue Type: Bug
Reporter: Elliott Clark
Assignee: Rajesh Nishtala
{code}
Thread 4139
(regionserver/hbase412.example.com/10.158.6.53:60020-splits-1429003183537):
State: WAITING
Blocked count: 131
Waited count: 228
Waiting on
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@50714dc3
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1371)
org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1325)
org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:352)
org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:252)
org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:509)
org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:84)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13467) Prototype using GRPC as IPC mechanism
[ https://issues.apache.org/jira/browse/HBASE-13467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495484#comment-14495484 ] Louis Ryan commented on HBASE-13467: Yes. I suspect that running both interfaces at once will be necessary. Haven't gotten to the security side of things yet. Once I've gotten plaintext working Ill take a look Prototype using GRPC as IPC mechanism - Key: HBASE-13467 URL: https://issues.apache.org/jira/browse/HBASE-13467 Project: HBase Issue Type: Improvement Components: API Affects Versions: 2.0.0 Reporter: Louis Ryan Priority: Minor GRPC provide an RPC layer for protocol buffers on top of Netty 4/5. This could be used to replace the current internal implementation. GRPC supports some advanced features like streaming, async, flow-control, cancellation timeout which might be useful Will prototype on GitHub here if folks are interested https://github.com/louiscryan/hbase -- This message was sent by Atlassian JIRA (v6.3.4#6332)
