[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927540#comment-15927540 ] ramkrishna.s.vasudevan commented on HBASE-16438: bq.ByteBufferChunkCell should be in hbase-server beside the MSLAB rather than out in hbase-common? It is a server-side only thing? Ditto with copyToChunkCell and ChunkCell? What you think? Yes I agree. bq.private long id = -1; It is passed on Construction. Can it change during lifetime of the Cell? Can make it final. It won't change. It is id only not an offset. Just an unique number. bq.Will it ever be the case that Cells from Chunks are persisted across restarts? Say, in a bucketcache that is persisted? Just wondering if the chunkid needs to be unique across restarts? I don't think so. We want any chunk's life time to be available till the segment having the chunk is flushed. So even if it is from pool we need to know the chunkId till the flush happens. bq.We have MSLABChunkCreator. So a Chunk is a 'piece' of a MSLAB? And a 'MSLABChunkCreateor' creates chunks or allocates pieces of the MSLAB? Do we have to have MSLAB in the name? Is it MSLAB only? I like the last question . MSLAB is created per segment and each MSLAB can have more than one Chunk. So it is not one to one here. So now we try to designate a ChunkCreator who does the creation and management of these chunks. bq.We keep a chunkIdMap? Is this of all chunks? How many chunks will there be? All threads will be banging on this Map? In case of ChunkPool we are limited to the pool size. but if there is no chunkPool then I think we will be having more number of chunks created on demand. That is why we have the logic of removing these chunks Ids from the map when there is no pool when we do the close() of a segment so that we are sure that we no longer need those chunkIds. In case of pool we cannot do that as we reuse the chunks. bq.Regards the below, who sets forceOnHeapOnly? Rather should we pass in the Cell and let the allocator figure where to allocate the memory? >From the Cell we cannot tell because in case of ChunkPool once we run out of >its max size the MSLAB decides where to create the chunk and not the cell. bq.So if no chunk pool, we keep chunk ids in a map elsewhere than in MSLABChunkCreateor? am not sure if you have seen [~anastas]'s concern. Infact I thought one MSLABChunkCreator is enought and we can pass the singleton ref with MSLAB and the pool. So that chunkCreator decides on chunkCreation but the responsibility is with MSLAB to either as the pool or chunkCreator directly. But she feels that is not right and better to refactor fully in such a way that only ChunkCreator does chunkCreation and init(which is the costly one) and with that change all the CAS operation and the way it works out. > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-16438_1.patch, HBASE-16438.patch, > MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927525#comment-15927525 ] ramkrishna.s.vasudevan commented on HBASE-16438: bq.I'll continue with the Allocator/Creator and all the rest... What do you think? Oh I see. I already did some work here to impl as per your suggestion. If you want to continue I can reduce the scope of this patch. I think lock unlock is needed along with CAS for the curChunk thing. And also if you see the other threds who is waiting for the curChunk though may get it but has to wait for the alloc to happen and then only it will succeed. I think so overall the same will happen here but only thing is the lock/unlock way may have some internal implication. > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-16438_1.patch, HBASE-16438.patch, > MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17584) Expose ScanMetrics with ResultScanner rather than Scan
[ https://issues.apache.org/jira/browse/HBASE-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927524#comment-15927524 ] Duo Zhang commented on HBASE-17584: --- Done. [~stack] [~enis]. > Expose ScanMetrics with ResultScanner rather than Scan > -- > > Key: HBASE-17584 > URL: https://issues.apache.org/jira/browse/HBASE-17584 > Project: HBase > Issue Type: Sub-task > Components: Client, mapreduce, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17584.patch, HBASE-17584-v1.patch > > > I think this have been discussed many times... It is a bad practice to > directly modify the Scan object passed in when calling getScanner. The reason > that we can not use a copy is we need to use the Scan object to expose scan > metrics. So we need to find another way to expose the metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927523#comment-15927523
]
stack commented on HBASE-16438:
---
I took a quick look at the patch.
ByteBufferChunkCell should be in hbase-server beside the MSLAB rather than out
in hbase-common? It is a server-side only thing? Ditto with copyToChunkCell and
ChunkCell? What you think?
NoTagByteBufferChunkCell <= You know what I'm going to say (smile).
Should this be final:
private long id = -1;
It is passed on Construction. Can it change during lifetime of the Cell?
Is it an 'id' or is it an 'offset'?
Can we pass in the id when we init a chunk?
Will it ever be the case that Cells from Chunks are persisted across restarts?
Say, in a bucketcache that is persisted? Just wondering if the chunkid needs to
be unique across restarts?
What is the nomenclature here?
We have MSLABChunkCreator. So a Chunk is a 'piece' of a MSLAB? And a
'MSLABChunkCreateor' creates chunks or allocates pieces of the MSLAB? Do we
have to have MSLAB in the name? Is it MSLAB only?
We keep a chunkIdMap? Is this of all chunks? How many chunks will there be? All
threads will be banging on this Map?
Regards the below, who sets forceOnHeapOnly? Rather should we pass in the Cell
and let the allocator figure where to allocate the memory?
Chunk createChunk(boolean forceOnheapOnly) {
So if no chunk pool, we keep chunk ids in a map elsewhere than in
MSLABChunkCreateor? Should we do this accounting in one place only?
Otherwise, looking good.
> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-16438_1.patch, HBASE-16438.patch,
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch,
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was
> created, the id of the chunk be embedded in it so that when doing flattening
> we can use the chunk id as a meta data. More details will follow once the
> initial tasks are completed.
> Why we need to embed the chunkid in the Cell is described by [~anastas] in
> this remark over in parent issue
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17790) Mark ReplicationAdmin's peerAdded and listReplicationPeers as Deprecated
[
https://issues.apache.org/jira/browse/HBASE-17790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927522#comment-15927522
]
Ashish Singhi commented on HBASE-17790:
---
{code} @deprecated user {@link
org.apache.hadoop.hbase.client.Admin#listReplicationPeers()} instead {code}
user -> use.
You can change it on commit. +1
> Mark ReplicationAdmin's peerAdded and listReplicationPeers as Deprecated
>
>
> Key: HBASE-17790
> URL: https://issues.apache.org/jira/browse/HBASE-17790
> Project: HBase
> Issue Type: Sub-task
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17790.master.001.patch,
> HBASE-17790.master.002.patch
>
>
> Now most of public method in ReplicationAdmin have been moved to Admin and
> marked as Deprecated. And peerAdded and listReplicationPeers method need be
> marked as Deprecated, too.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17584) Expose ScanMetrics with ResultScanner rather than Scan
[ https://issues.apache.org/jira/browse/HBASE-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927513#comment-15927513 ] Duo Zhang commented on HBASE-17584: --- OK, let me start a thread on mailing list. > Expose ScanMetrics with ResultScanner rather than Scan > -- > > Key: HBASE-17584 > URL: https://issues.apache.org/jira/browse/HBASE-17584 > Project: HBase > Issue Type: Sub-task > Components: Client, mapreduce, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17584.patch, HBASE-17584-v1.patch > > > I think this have been discussed many times... It is a bad practice to > directly modify the Scan object passed in when calling getScanner. The reason > that we can not use a copy is we need to use the Scan object to expose scan > metrics. So we need to find another way to expose the metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17790) Mark ReplicationAdmin's peerAdded and listReplicationPeers as Deprecated
[
https://issues.apache.org/jira/browse/HBASE-17790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927511#comment-15927511
]
Hadoop QA commented on HBASE-17790:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 24s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 9s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 103m 39s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 149m 49s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12858996/HBASE-17790.master.002.patch
|
| JIRA Issue | HBASE-17790 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux c4e5eebc8c1b 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 1849e8a |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6112/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6112/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6112/console |
| Powered by | Apache Yetus 0.3.0
[jira] [Commented] (HBASE-17789) ZKConfig.getZKQuorumServersStringFromHbaseConfig creates a incorrect zkquorum string
[ https://issues.apache.org/jira/browse/HBASE-17789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927507#comment-15927507 ] Andrew Purtell commented on HBASE-17789: That behavior has changed for invalid input doesn't absolve the user of the API of proper use. ZkConfig is not a public API. You shouldn't be using it [~vishk] . Your app should do its own string handling. > ZKConfig.getZKQuorumServersStringFromHbaseConfig creates a incorrect zkquorum > string > > > Key: HBASE-17789 > URL: https://issues.apache.org/jira/browse/HBASE-17789 > Project: HBase > Issue Type: Bug >Reporter: Vishal Khandelwal >Assignee: Vishal Khandelwal > > , if we try to a hbase using "HBaseAdmin" by reading a value of > ""hbase.zookeeper.quorum" from list peers with value as > "a:2181,b:2181c:2181:/hbase". > org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersStringFromHbaseConfig > returns a incorrect queryString which causes following exception Because > last host would see :/hbase also in the string. This method was existing > earlier but never being called before above change -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17584) Expose ScanMetrics with ResultScanner rather than Scan
[ https://issues.apache.org/jira/browse/HBASE-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927506#comment-15927506 ] stack commented on HBASE-17584: --- [~enis] [~Apache9] Duo (or I), lets start a thread on mailing list w/ @enis statement looking for agreement (makes sense to me) I'll update the refguide after the mailing list discussion is done. > Expose ScanMetrics with ResultScanner rather than Scan > -- > > Key: HBASE-17584 > URL: https://issues.apache.org/jira/browse/HBASE-17584 > Project: HBase > Issue Type: Sub-task > Components: Client, mapreduce, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17584.patch, HBASE-17584-v1.patch > > > I think this have been discussed many times... It is a bad practice to > directly modify the Scan object passed in when calling getScanner. The reason > that we can not use a copy is we need to use the Scan object to expose scan > metrics. So we need to find another way to expose the metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17453) add Ping into HBase server for deprecated GetProtocolVersion
[ https://issues.apache.org/jira/browse/HBASE-17453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927498#comment-15927498 ] stack commented on HBASE-17453: --- [~tychang] This strikes me as an Admin function. Asyncclient will not be able to use it if it is in Admin? On CP, you make a good point. > add Ping into HBase server for deprecated GetProtocolVersion > > > Key: HBASE-17453 > URL: https://issues.apache.org/jira/browse/HBASE-17453 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 1.2.2 >Reporter: Tianying Chang >Assignee: Tianying Chang >Priority: Minor > Fix For: 2.0.0, 1.2.5 > > Attachments: HBASE-17453-1.2.patch, HBASE-17453-master.patch, > HBASE-17453-master-v1.patch, HBASE-17453-master-v2.patch > > > Our HBase service is hosted in AWS. We saw cases where the connection between > the client (Asynchbase in our case) and server stop working but did not throw > any exception, therefore traffic stuck. So we added a "Ping" feature in > AsyncHBase 1.5 by utilizing the GetProtocolVersion() API provided at RS side, > if no traffic for given time, we send the "Ping", if no response back for > "Ping", we assume the connect is bad and reconnect. > Now we are upgrading cluster from 94 to 1.2. However, GetProtocolVersion() is > deprecated. To be able to support same detect/reconnect feature, we added > Ping() in our internal HBase 1.2 branch, and also patched accordingly in > Asynchbase 1.7. > We would like to open source this feature since it is useful for use case in > AWS environment. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17792) Use a shared thread pool for AtomicityWriter, AtomicGetReader, AtomicScanReader's connections in TestAcidGuarantees
[ https://issues.apache.org/jira/browse/HBASE-17792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927492#comment-15927492 ] stack commented on HBASE-17792: --- lgtm [~huaxiang] Did we always make lots of threads in this test or was there a recent change that brought it on? Thanks boss. > Use a shared thread pool for AtomicityWriter, AtomicGetReader, > AtomicScanReader's connections in TestAcidGuarantees > --- > > Key: HBASE-17792 > URL: https://issues.apache.org/jira/browse/HBASE-17792 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: HBASE-17792-master-001.patch > > > AtomicityWriter, AtomicGetReader, AtomicScanReader creates connection inside, > since the thread pool is not shared, by default each connection will create > 256 threads. If there are 5 AtomicityWriters, 1 AtomicGetReader, and 1 > AtomicScanReader, it will create lots of threads and causes max user > processes to be reached, and OOME. Use a share thread pool to work around > this issue. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17790) Mark ReplicationAdmin's peerAdded and listReplicationPeers as Deprecated
[
https://issues.apache.org/jira/browse/HBASE-17790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927450#comment-15927450
]
Hadoop QA commented on HBASE-17790:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 41s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 24s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 106m 23s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 158m 17s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12858859/HBASE-17790.master.001.patch
|
| JIRA Issue | HBASE-17790 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux ba53a0945eb8 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0ecb678 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6111/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6111/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Mark ReplicationAdmin's peerAdded and
[jira] [Commented] (HBASE-17754) [C++] RawAsyncTable
[
https://issues.apache.org/jira/browse/HBASE-17754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927437#comment-15927437
]
Enis Soztutar commented on HBASE-17754:
---
bq. We should keep the unit test. client-test is like system test.
The test is back in v3. Some changes were needed to remove some unneeded
mocking. The MockAsyncConnection turned out to be causing problems with the
linker when extending the base interface, so I've ended up extending
AsyncConncetionImpl instead.
bq. Why do we need the std::pair? Can we simplify this?
Ok, sorry for the confusion. I had added that as a quick way to test the theory
that the Caller goes out of scope, so the Promise object that the Caller owns
is deallocated. This causes the "broken promise" result that we were debugging
earlier. Turns out that the theory is right that since we are capturing by
{{[&,this]}} in almost all of the lambda in the Caller implementation, the
Caller object should not be deallocated before all of the lamdbas are executed.
Now, when RawAsyncTable creates the caller object, and returns the Future, we
are still injecting the Caller into the Future so that we will keep a reference
around.
For 3 and 4, offline conversation, the patch seems correct.
> [C++] RawAsyncTable
> ---
>
> Key: HBASE-17754
> URL: https://issues.apache.org/jira/browse/HBASE-17754
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: hbase-17754_v0.patch, hbase-17754_v1.patch,
> hbase-17754-v2.patch, hbase-17754-v3.patch
>
>
> We need RawAsyncTable to connect {{table.h}} to the async retrying callers.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17754) [C++] RawAsyncTable
[
https://issues.apache.org/jira/browse/HBASE-17754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-17754:
--
Attachment: hbase-17754-v3.patch
> [C++] RawAsyncTable
> ---
>
> Key: HBASE-17754
> URL: https://issues.apache.org/jira/browse/HBASE-17754
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: hbase-17754_v0.patch, hbase-17754_v1.patch,
> hbase-17754-v2.patch, hbase-17754-v3.patch
>
>
> We need RawAsyncTable to connect {{table.h}} to the async retrying callers.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HBASE-17003) Update book for filesystem use quotas
[ https://issues.apache.org/jira/browse/HBASE-17003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser reassigned HBASE-17003: -- Assignee: Josh Elser > Update book for filesystem use quotas > - > > Key: HBASE-17003 > URL: https://issues.apache.org/jira/browse/HBASE-17003 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Josh Elser >Assignee: Josh Elser > > Need to make sure that the book's section on quotas reflect the final > implementation of the filesystem-use quota support. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17707) New More Accurate Table Skew cost function/generator
[ https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927426#comment-15927426 ] Ted Yu commented on HBASE-17707: Integrated to master branch. Keeping fingers crossed. > New More Accurate Table Skew cost function/generator > > > Key: HBASE-17707 > URL: https://issues.apache.org/jira/browse/HBASE-17707 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Kahlil Oppenheimer >Assignee: Kahlil Oppenheimer >Priority: Minor > Fix For: 2.0 > > Attachments: HBASE-17707-00.patch, HBASE-17707-01.patch, > HBASE-17707-02.patch, HBASE-17707-03.patch, HBASE-17707-04.patch, > HBASE-17707-05.patch, HBASE-17707-06.patch, HBASE-17707-07.patch, > HBASE-17707-08.patch, HBASE-17707-09.patch, test-balancer2-13617.out > > > This patch includes new version of the TableSkewCostFunction and a new > TableSkewCandidateGenerator. > The new TableSkewCostFunction computes table skew by counting the minimal > number of region moves required for a given table to perfectly balance the > table across the cluster (i.e. as if the regions from that table had been > round-robin-ed across the cluster). This number of moves is computer for each > table, then normalized to a score between 0-1 by dividing by the number of > moves required in the absolute worst case (i.e. the entire table is stored on > one server), and stored in an array. The cost function then takes a weighted > average of the average and maximum value across all tables. The weights in > this average are configurable to allow for certain users to more strongly > penalize situations where one table is skewed versus where every table is a > little bit skewed. To better spread this value more evenly across the range > 0-1, we take the square root of the weighted average to get the final value. > The new TableSkewCandidateGenerator generates region moves/swaps to optimize > the above TableSkewCostFunction. It first simply tries to move regions until > each server has the right number of regions, then it swaps regions around > such that each region swap improves table skew across the cluster. > We tested the cost function and generator in our production clusters with > 100s of TBs of data and 100s of tables across dozens of servers and found > both to be very performant and accurate. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17789) ZKConfig.getZKQuorumServersStringFromHbaseConfig creates a incorrect zkquorum string
[ https://issues.apache.org/jira/browse/HBASE-17789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927419#comment-15927419 ] Vishal Khandelwal commented on HBASE-17789: --- [~elserj] : You may be right that app should handle this but this is the behaviour change from 0.98.23 to 0.98.24. till 0.98.23 same code use to work. IMO its not does 1 app may impact other apps which might be making this assumption. For us also it started breakin in 0.98.24 > ZKConfig.getZKQuorumServersStringFromHbaseConfig creates a incorrect zkquorum > string > > > Key: HBASE-17789 > URL: https://issues.apache.org/jira/browse/HBASE-17789 > Project: HBase > Issue Type: Bug >Reporter: Vishal Khandelwal >Assignee: Vishal Khandelwal > > , if we try to a hbase using "HBaseAdmin" by reading a value of > ""hbase.zookeeper.quorum" from list peers with value as > "a:2181,b:2181c:2181:/hbase". > org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersStringFromHbaseConfig > returns a incorrect queryString which causes following exception Because > last host would see :/hbase also in the string. This method was existing > earlier but never being called before above change -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17790) Mark ReplicationAdmin's peerAdded and listReplicationPeers as Deprecated
[ https://issues.apache.org/jira/browse/HBASE-17790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927358#comment-15927358 ] Guanghao Zhang commented on HBASE-17790: These methods are marked as @VisibleForTesting too. And peerAdded is not used anymore. So only add "use instead" to listReplicationPeers method. Thanks. > Mark ReplicationAdmin's peerAdded and listReplicationPeers as Deprecated > > > Key: HBASE-17790 > URL: https://issues.apache.org/jira/browse/HBASE-17790 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17790.master.001.patch, > HBASE-17790.master.002.patch > > > Now most of public method in ReplicationAdmin have been moved to Admin and > marked as Deprecated. And peerAdded and listReplicationPeers method need be > marked as Deprecated, too. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17790) Mark ReplicationAdmin's peerAdded and listReplicationPeers as Deprecated
[ https://issues.apache.org/jira/browse/HBASE-17790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17790: --- Attachment: HBASE-17790.master.002.patch > Mark ReplicationAdmin's peerAdded and listReplicationPeers as Deprecated > > > Key: HBASE-17790 > URL: https://issues.apache.org/jira/browse/HBASE-17790 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17790.master.001.patch, > HBASE-17790.master.002.patch > > > Now most of public method in ReplicationAdmin have been moved to Admin and > marked as Deprecated. And peerAdded and listReplicationPeers method need be > marked as Deprecated, too. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17792) Use a shared thread pool for AtomicityWriter, AtomicGetReader, AtomicScanReader's connections in TestAcidGuarantees
[
https://issues.apache.org/jira/browse/HBASE-17792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927348#comment-15927348
]
Hadoop QA commented on HBASE-17792:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 47s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 100m 10s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 143m 51s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12858976/HBASE-17792-master-001.patch
|
| JIRA Issue | HBASE-17792 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux a60fa42b6145 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0ecb678 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6110/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6110/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Use a shared thread pool for AtomicityWriter, AtomicGetReader,
> AtomicScanReader's connections in TestAcidGuarantees
> ---
>
> Key: HBASE-17792
> URL: https://issues.apache.org/jira/browse/HBASE-17792
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>
[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927306#comment-15927306 ] chunhui shen commented on HBASE-15314: -- [~ram_krish] Please help to commit it, thanks :) > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, > HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, > HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, > HBASE-15314-v6.patch, HBASE-15314-v7.patch, HBASE-15314-v8.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17426) Inconsistent environment variable names for enabling JMX
[
https://issues.apache.org/jira/browse/HBASE-17426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927305#comment-15927305
]
Ted Yu commented on HBASE-17426:
lgtm
> Inconsistent environment variable names for enabling JMX
>
>
> Key: HBASE-17426
> URL: https://issues.apache.org/jira/browse/HBASE-17426
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Chia-Ping Tsai
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17426.v0.patch
>
>
> In conf/hbase-env.sh :
> {code}
> # export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false"
> # If you want to configure BucketCache, specify '-XX: MaxDirectMemorySize='
> with proper direct memory size
> # export HBASE_THRIFT_OPTS="$HBASE_JMX_BASE
> -Dcom.sun.management.jmxremote.port=10103"
> {code}
> But in bin/hbase-config.sh , a different variable is used:
> {code}
> # Thrift JMX opts
> if [ -n "$HBASE_JMX_OPTS" ] && [ -z "$HBASE_THRIFT_JMX_OPTS" ]; then
> HBASE_THRIFT_JMX_OPTS="$HBASE_JMX_OPTS
> -Dcom.sun.management.jmxremote.port=10103"
> fi
> {code}
> The variable names should be aligned for better user experience.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17740) Correct the semantic of batch and partial for async client
[ https://issues.apache.org/jira/browse/HBASE-17740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17740: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master. Thanks [~yangzhe1991] for reviewing. > Correct the semantic of batch and partial for async client > -- > > Key: HBASE-17740 > URL: https://issues.apache.org/jira/browse/HBASE-17740 > Project: HBase > Issue Type: Sub-task > Components: asyncclient, Client, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17740.patch, HBASE-17740-v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17793) Backport ScanResultCache related code to branch-1
Duo Zhang created HBASE-17793: - Summary: Backport ScanResultCache related code to branch-1 Key: HBASE-17793 URL: https://issues.apache.org/jira/browse/HBASE-17793 Project: HBase Issue Type: Sub-task Components: Client, scan Affects Versions: 1.4.0 Reporter: Duo Zhang Assignee: Duo Zhang Fix For: 1.4.0 To simplity our code, and also keep the same code base for master and branch-1. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17740) Correct the semantic of batch and partial for async client
[ https://issues.apache.org/jira/browse/HBASE-17740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927293#comment-15927293 ] Duo Zhang commented on HBASE-17740: --- OK, all green. Will commit shortly. > Correct the semantic of batch and partial for async client > -- > > Key: HBASE-17740 > URL: https://issues.apache.org/jira/browse/HBASE-17740 > Project: HBase > Issue Type: Sub-task > Components: asyncclient, Client, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17740.patch, HBASE-17740-v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17758) [RSGROUP] Add shell command to move servers and tables at the same time
[
https://issues.apache.org/jira/browse/HBASE-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927292#comment-15927292
]
Andrew Purtell commented on HBASE-17758:
I'd like to get this in but looks like there are still test failures
> [RSGROUP] Add shell command to move servers and tables at the same time
> ---
>
> Key: HBASE-17758
> URL: https://issues.apache.org/jira/browse/HBASE-17758
> Project: HBase
> Issue Type: New Feature
> Components: rsgroup
>Affects Versions: 2.0.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
> Attachments: HBASE-17758-v1.patch, HBASE-17758-v2.patch,
> HBASE-17758-v3.patch, HBASE-17758-v4.patch
>
>
> Currently add a new group perform the following steps:
> {code:javascript}
> hbase(main):001:0> add_rsgroup 'mygroup'
> Took 0.3840 seconds
> hbase(main):002:0> move_servers_rsgroup
> 'mygroup',['hbase-rs-01:16030','hbase-rs-02:16030']
> Took 3.5040 seconds
> hbase(main):003:0> move_tables_rsgroup 'mygroup',['example']
> Took 0.2710 seconds
> {code}
> 1. move_servers_rsgroup will unassign all regions on hbase-rs-01 and
> hbase-rs-02
> 2. move_tables_rsgroup will also reassign all the regions of the table
> example.
> This will lead to a large number of regions to migrate and affect the data
> locality.
> However, some regions of the table that are distributed on hbase-rs-01 and
> hbase-rs-02, do not need to be moved.
> So,we need a new shell command *move_servers_tables_rsgroup* which minimizes
> the number of regions needed to move.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16322) Disable filter for raw scan
[ https://issues.apache.org/jira/browse/HBASE-16322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927276#comment-15927276 ] Duo Zhang commented on HBASE-16322: --- I think these filters are private and only used at server side? The point here is that, the current filter is not designed to deal with delete marker, and it is too powerful for our server side usage. I'm not familiar with AccessControlFilter and VisibilityLabelFilter but I guess cell level filter is enough for them? There is a issue for introducing a new filter that can only be used at server side, HBASE-16323. Thanks. > Disable filter for raw scan > --- > > Key: HBASE-16322 > URL: https://issues.apache.org/jira/browse/HBASE-16322 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang > > As we will pass delete markers to the filter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17790) Mark ReplicationAdmin's peerAdded and listReplicationPeers as Deprecated
[ https://issues.apache.org/jira/browse/HBASE-17790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17790: --- Status: Patch Available (was: Open) > Mark ReplicationAdmin's peerAdded and listReplicationPeers as Deprecated > > > Key: HBASE-17790 > URL: https://issues.apache.org/jira/browse/HBASE-17790 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17790.master.001.patch > > > Now most of public method in ReplicationAdmin have been moved to Admin and > marked as Deprecated. And peerAdded and listReplicationPeers method need be > marked as Deprecated, too. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17426) Inconsistent environment variable names for enabling JMX
[
https://issues.apache.org/jira/browse/HBASE-17426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927230#comment-15927230
]
Hadoop QA commented on HBASE-17426:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 7s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
5s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 0s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 42s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12858969/HBASE-17426.v0.patch |
| JIRA Issue | HBASE-17426 |
| Optional Tests | asflicense shellcheck shelldocs |
| uname | Linux a644fcde7643 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0ecb678 |
| shellcheck | v0.4.5 |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6109/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Inconsistent environment variable names for enabling JMX
>
>
> Key: HBASE-17426
> URL: https://issues.apache.org/jira/browse/HBASE-17426
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Chia-Ping Tsai
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17426.v0.patch
>
>
> In conf/hbase-env.sh :
> {code}
> # export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false"
> # If you want to configure BucketCache, specify '-XX: MaxDirectMemorySize='
> with proper direct memory size
> # export HBASE_THRIFT_OPTS="$HBASE_JMX_BASE
> -Dcom.sun.management.jmxremote.port=10103"
> {code}
> But in bin/hbase-config.sh , a different variable is used:
> {code}
> # Thrift JMX opts
> if [ -n "$HBASE_JMX_OPTS" ] && [ -z "$HBASE_THRIFT_JMX_OPTS" ]; then
> HBASE_THRIFT_JMX_OPTS="$HBASE_JMX_OPTS
> -Dcom.sun.management.jmxremote.port=10103"
> fi
> {code}
> The variable names should be aligned for better user experience.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17723) ClientAsyncPrefetchScanner may end prematurely when the size of the cache is one
[ https://issues.apache.org/jira/browse/HBASE-17723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927225#comment-15927225 ] Hudson commented on HBASE-17723: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2679 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2679/]) HBASE-17723 ClientAsyncPrefetchScanner may end prematurely when the size (chia7712: rev 0ecb6782593039af75a45c25481f1dbf7cbd6928) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestScannersFromClientSide.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientAsyncPrefetchScanner.java > ClientAsyncPrefetchScanner may end prematurely when the size of the cache is > one > > > Key: HBASE-17723 > URL: https://issues.apache.org/jira/browse/HBASE-17723 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 > Environment: Ubuntu 14.04 LTS / x86_64 / 16GB RAM >Reporter: Anup Halarnkar >Assignee: Chia-Ping Tsai > Fix For: 2.0.0 > > Attachments: HBASE-17723.v0.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch, HBASE-17723.v1.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch > > > Command to build and test: > mvn clean install -X > JAVA_TOOL_OPTIONS: -Xmx4G -XX:MaxPermSize=4G -Xms1G > Failure in tests: > Results : > Failed tests: > > TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync:161->testRowsSeenMetric:174->testRowsSeenMetric:204->testMetric:330 > Metric: ROWS_SCANNED Expected: 9 Actual: 8 expected:<9> but was:<8> > > TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing:78->testReplayWorksWithMemoryCompactionPolicy:149->verifyAllEditsMadeItIn:195 > Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > Tests run: 1761, Failures: 2, Errors: 0, Skipped: 6 > > This is the output for 1st test failure: > --- > Test set: org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > --- > Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.971 sec <<< > FAILURE! - in org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > testRowsSeenMetricWithAsync(org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide) > Time elapsed: 1.134 sec <<< FAILURE! > java.lang.AssertionError: Metric: ROWS_SCANNED Expected: 9 Actual: 8 > expected:<9> but was:<8> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testMetric(TestServerSideScanMetricsFromClientSide.java:330) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:204) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:174) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync(TestServerSideScanMetricsFromClientSide.java:161) > > This is output of 2nd test failure: > --- > Test set: org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > --- > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 11.911 sec > <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > testReplayWorksThoughLotsOfFlushing(org.apache.hadoop.hbase.regionserver.TestRecoveredEdits) > Time elapsed: 11.871 sec <<< FAILURE! > java.lang.AssertionError: Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.verifyAllEditsMadeItIn(TestRecoveredEdits.java:195) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksWithMemoryCompactionPolicy(TestRecoveredEdits.java:149) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing(TestRecoveredEdits.java:78) > Please let me know if you need more information > Thanks, > Anup -- This message was sent by
[jira] [Updated] (HBASE-17792) Use a shared thread pool for AtomicityWriter, AtomicGetReader, AtomicScanReader's connections in TestAcidGuarantees
[ https://issues.apache.org/jira/browse/HBASE-17792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-17792: - Status: Patch Available (was: Open) > Use a shared thread pool for AtomicityWriter, AtomicGetReader, > AtomicScanReader's connections in TestAcidGuarantees > --- > > Key: HBASE-17792 > URL: https://issues.apache.org/jira/browse/HBASE-17792 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: HBASE-17792-master-001.patch > > > AtomicityWriter, AtomicGetReader, AtomicScanReader creates connection inside, > since the thread pool is not shared, by default each connection will create > 256 threads. If there are 5 AtomicityWriters, 1 AtomicGetReader, and 1 > AtomicScanReader, it will create lots of threads and causes max user > processes to be reached, and OOME. Use a share thread pool to work around > this issue. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17792) Use a shared thread pool for AtomicityWriter, AtomicGetReader, AtomicScanReader's connections in TestAcidGuarantees
[ https://issues.apache.org/jira/browse/HBASE-17792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-17792: - Attachment: HBASE-17792-master-001.patch > Use a shared thread pool for AtomicityWriter, AtomicGetReader, > AtomicScanReader's connections in TestAcidGuarantees > --- > > Key: HBASE-17792 > URL: https://issues.apache.org/jira/browse/HBASE-17792 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: HBASE-17792-master-001.patch > > > AtomicityWriter, AtomicGetReader, AtomicScanReader creates connection inside, > since the thread pool is not shared, by default each connection will create > 256 threads. If there are 5 AtomicityWriters, 1 AtomicGetReader, and 1 > AtomicScanReader, it will create lots of threads and causes max user > processes to be reached, and OOME. Use a share thread pool to work around > this issue. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17792) Use a shared thread pool for AtomicityWriter, AtomicGetReader, AtomicScanReader's connections in TestAcidGuarantees
huaxiang sun created HBASE-17792: Summary: Use a shared thread pool for AtomicityWriter, AtomicGetReader, AtomicScanReader's connections in TestAcidGuarantees Key: HBASE-17792 URL: https://issues.apache.org/jira/browse/HBASE-17792 Project: HBase Issue Type: Bug Affects Versions: 2.0.0 Reporter: huaxiang sun Assignee: huaxiang sun Priority: Minor AtomicityWriter, AtomicGetReader, AtomicScanReader creates connection inside, since the thread pool is not shared, by default each connection will create 256 threads. If there are 5 AtomicityWriters, 1 AtomicGetReader, and 1 AtomicScanReader, it will create lots of threads and causes max user processes to be reached, and OOME. Use a share thread pool to work around this issue. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17426) Inconsistent environment variable names for enabling JMX
[
https://issues.apache.org/jira/browse/HBASE-17426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-17426:
---
Assignee: Chia-Ping Tsai
Status: Patch Available (was: Open)
> Inconsistent environment variable names for enabling JMX
>
>
> Key: HBASE-17426
> URL: https://issues.apache.org/jira/browse/HBASE-17426
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Chia-Ping Tsai
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17426.v0.patch
>
>
> In conf/hbase-env.sh :
> {code}
> # export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false"
> # If you want to configure BucketCache, specify '-XX: MaxDirectMemorySize='
> with proper direct memory size
> # export HBASE_THRIFT_OPTS="$HBASE_JMX_BASE
> -Dcom.sun.management.jmxremote.port=10103"
> {code}
> But in bin/hbase-config.sh , a different variable is used:
> {code}
> # Thrift JMX opts
> if [ -n "$HBASE_JMX_OPTS" ] && [ -z "$HBASE_THRIFT_JMX_OPTS" ]; then
> HBASE_THRIFT_JMX_OPTS="$HBASE_JMX_OPTS
> -Dcom.sun.management.jmxremote.port=10103"
> fi
> {code}
> The variable names should be aligned for better user experience.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17426) Inconsistent environment variable names for enabling JMX
[
https://issues.apache.org/jira/browse/HBASE-17426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-17426:
---
Attachment: HBASE-17426.v0.patch
a trivial patch
> Inconsistent environment variable names for enabling JMX
>
>
> Key: HBASE-17426
> URL: https://issues.apache.org/jira/browse/HBASE-17426
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17426.v0.patch
>
>
> In conf/hbase-env.sh :
> {code}
> # export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false"
> # If you want to configure BucketCache, specify '-XX: MaxDirectMemorySize='
> with proper direct memory size
> # export HBASE_THRIFT_OPTS="$HBASE_JMX_BASE
> -Dcom.sun.management.jmxremote.port=10103"
> {code}
> But in bin/hbase-config.sh , a different variable is used:
> {code}
> # Thrift JMX opts
> if [ -n "$HBASE_JMX_OPTS" ] && [ -z "$HBASE_THRIFT_JMX_OPTS" ]; then
> HBASE_THRIFT_JMX_OPTS="$HBASE_JMX_OPTS
> -Dcom.sun.management.jmxremote.port=10103"
> fi
> {code}
> The variable names should be aligned for better user experience.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17426) Inconsistent environment variable names for enabling JMX
[
https://issues.apache.org/jira/browse/HBASE-17426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-17426:
---
Fix Version/s: 1.4.0
2.0.0
> Inconsistent environment variable names for enabling JMX
>
>
> Key: HBASE-17426
> URL: https://issues.apache.org/jira/browse/HBASE-17426
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17426.v0.patch
>
>
> In conf/hbase-env.sh :
> {code}
> # export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false"
> # If you want to configure BucketCache, specify '-XX: MaxDirectMemorySize='
> with proper direct memory size
> # export HBASE_THRIFT_OPTS="$HBASE_JMX_BASE
> -Dcom.sun.management.jmxremote.port=10103"
> {code}
> But in bin/hbase-config.sh , a different variable is used:
> {code}
> # Thrift JMX opts
> if [ -n "$HBASE_JMX_OPTS" ] && [ -z "$HBASE_THRIFT_JMX_OPTS" ]; then
> HBASE_THRIFT_JMX_OPTS="$HBASE_JMX_OPTS
> -Dcom.sun.management.jmxremote.port=10103"
> fi
> {code}
> The variable names should be aligned for better user experience.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16247) SparkSQL Avro serialization doesn't handle enums correctly
[ https://issues.apache.org/jira/browse/HBASE-16247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927158#comment-15927158 ] Jerry He commented on HBASE-16247: -- Looking at the hbase spark code. It does not seem to be a problem because hbase spark sql does not construct a Record from Avro type ENUM directly if I look at it right. > SparkSQL Avro serialization doesn't handle enums correctly > -- > > Key: HBASE-16247 > URL: https://issues.apache.org/jira/browse/HBASE-16247 > Project: HBase > Issue Type: Bug > Components: spark >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Priority: Critical > Fix For: 2.0.0 > > > Avro's generic api expects GenericEnumSymbol as the runtime type for > instances of fields that are of Avro type ENUM. The Avro 1.7 libraries are > lax in some cases for handling this, but the 1.8 libraries are strict. We > should proactively fix our serialization. > (the lax serialization in 1.7 fails for some nested use in unions, see > AVRO-997 for details) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16247) SparkSQL Avro serialization doesn't handle enums correctly
[ https://issues.apache.org/jira/browse/HBASE-16247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927153#comment-15927153 ] Jerry He commented on HBASE-16247: -- Hi, [~busbey] Can you provide more details on the problem with hbase spark sql handling of the Avro enum type? > SparkSQL Avro serialization doesn't handle enums correctly > -- > > Key: HBASE-16247 > URL: https://issues.apache.org/jira/browse/HBASE-16247 > Project: HBase > Issue Type: Bug > Components: spark >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Priority: Critical > Fix For: 2.0.0 > > > Avro's generic api expects GenericEnumSymbol as the runtime type for > instances of fields that are of Avro type ENUM. The Avro 1.7 libraries are > lax in some cases for handling this, but the 1.8 libraries are strict. We > should proactively fix our serialization. > (the lax serialization in 1.7 fails for some nested use in unions, see > AVRO-997 for details) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16322) Disable filter for raw scan
[ https://issues.apache.org/jira/browse/HBASE-16322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927140#comment-15927140 ] Chia-Ping Tsai commented on HBASE-16322: Does the security still work if we disable the filter for raw scan ? How do we use the AccessControlFilter and VisibilityLabelFilter for the raw scan? > Disable filter for raw scan > --- > > Key: HBASE-16322 > URL: https://issues.apache.org/jira/browse/HBASE-16322 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang > > As we will pass delete markers to the filter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17754) [C++] RawAsyncTable
[
https://issues.apache.org/jira/browse/HBASE-17754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926857#comment-15926857
]
Enis Soztutar edited comment on HBASE-17754 at 3/15/17 10:07 PM:
-
Thanks for the new patch.
# We should keep the unit test. client-test is like system test.
# I was referring this
{noformat}
std::pair>,
Future> Get(const hbase::Get& get);
{noformat}
Why do we need the std::pair? Can we simplify this?
# This is the sequence of call
{noformat}
template
std::shared_ptr
RawAsyncTable::CreateCallerBuilder
{noformat}
returns builder, and builder->build() returns
AsyncSingleRequestRpcRetryingCaller,
then AsyncSingleRequestRpcRetryingCaller.call() returns RESP which is the
hbase::Result.
# you may need to do cast like this, std::shared_ptr itself is agnostic of
polymorphism of the enclosed type.
{noformat}
115 std::shared_ptr RpcClient::CreateRpcChannel(const
std::string ,
116 uint16_t port, std::shared_ptr ticket, int rpc_timeout) {
117
118 std::shared_ptr channel = std::make_shared<
119 RpcChannelImplementation>(shared_from_this(), host, port, ticket,
120 rpc_timeout);
121
122 /* static_pointer_cast is safe since RpcChannelImplementation derives
123* from RpcChannel, otherwise, dynamic_pointer_cast should be used. */
124 return std::static_pointer_cast(channel);
125 }
{noformat}
Otherwise, v2 looks fine.
was (Author: xiaobingo):
Thanks for the new patch.
# We should keep the unit test. client-test is like system test.
# I was referring this
{noformat}
std::pair>,
Future> Get(const hbase::Get& get);
{noformat}
Why do we need the std::pair? Can we simplify this?
# This is the sequence of call
{noformat}
template
std::shared_ptr
RawAsyncTable::CreateCallerBuilder
{noformat}
returns builder, and builder->build() returns
AsyncSingleRequestRpcRetryingCaller,
then AsyncSingleRequestRpcRetryingCaller.call() returns RESP which is the
hbase::Result.
{noformat}
# you may need to do cast like this, std::shared_ptr itself is agnostic of
polymorphism of the enclosed type.
115 std::shared_ptr RpcClient::CreateRpcChannel(const
std::string ,
116 uint16_t port, std::shared_ptr ticket, int rpc_timeout) {
117
118 std::shared_ptr channel = std::make_shared<
119 RpcChannelImplementation>(shared_from_this(), host, port, ticket,
120 rpc_timeout);
121
122 /* static_pointer_cast is safe since RpcChannelImplementation derives
123* from RpcChannel, otherwise, dynamic_pointer_cast should be used. */
124 return std::static_pointer_cast(channel);
125 }
Otherwise, v2 looks fine.
> [C++] RawAsyncTable
> ---
>
> Key: HBASE-17754
> URL: https://issues.apache.org/jira/browse/HBASE-17754
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: hbase-17754_v0.patch, hbase-17754_v1.patch,
> hbase-17754-v2.patch
>
>
> We need RawAsyncTable to connect {{table.h}} to the async retrying callers.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17771) [C++] Classes required for implementation of BatchCallerBuilder
[ https://issues.apache.org/jira/browse/HBASE-17771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudeep Sunthankar updated HBASE-17771: -- Attachment: HBASE-17771.HBASE-14850.v2.patch Have addressed the feedback in the last patch Thanks > [C++] Classes required for implementation of BatchCallerBuilder > --- > > Key: HBASE-17771 > URL: https://issues.apache.org/jira/browse/HBASE-17771 > Project: HBase > Issue Type: Sub-task >Reporter: Sudeep Sunthankar >Assignee: Sudeep Sunthankar > Fix For: HBASE-14850 > > Attachments: HBASE-17771.HBASE-14850.v1.patch, > HBASE-17771.HBASE-14850.v2.patch > > > Separating depedencies of BatchCallerBuilder. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17576) [C++] Implement request retry mechanism over RPC for Multi calls.
[ https://issues.apache.org/jira/browse/HBASE-17576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudeep Sunthankar updated HBASE-17576: -- Attachment: HBASE-17576.HBASE-14850.v6.patch v6 patch consisting of only AsyncBatchRpcRetryingCaller class > [C++] Implement request retry mechanism over RPC for Multi calls. > - > > Key: HBASE-17576 > URL: https://issues.apache.org/jira/browse/HBASE-17576 > Project: HBase > Issue Type: Sub-task >Reporter: Sudeep Sunthankar >Assignee: Sudeep Sunthankar > Attachments: HBASE-17576.HBASE-14850.v1.patch, > HBASE-17576.HBASE-14850.v2.patch, HBASE-17576.HBASE-14850.v3.patch, > HBASE-17576.HBASE-14850.v4.patch, HBASE-17576.HBASE-14850.v5.patch, > HBASE-17576.HBASE-14850.v6.patch > > > This work is based on top of HBASE-17465. Multi Calls will be based on this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17453) add Ping into HBase server for deprecated GetProtocolVersion
[
https://issues.apache.org/jira/browse/HBASE-17453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927070#comment-15927070
]
Hadoop QA commented on HBASE-17453:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 50s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 8s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 38s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 31s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 24s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 106m 34s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
55s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 182m 53s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12858928/HBASE-17453-master-v2.patch
|
| JIRA Issue | HBASE-17453 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile cc hbaseprotoc |
| uname | Linux 52c314abe577 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
[jira] [Created] (HBASE-17791) Locality should not be affected for non-faulty region servers at startup
Lars George created HBASE-17791:
---
Summary: Locality should not be affected for non-faulty region
servers at startup
Key: HBASE-17791
URL: https://issues.apache.org/jira/browse/HBASE-17791
Project: HBase
Issue Type: Improvement
Components: Balancer, Region Assignment
Affects Versions: 1.1.8, 1.2.4, 1.0.3, 1.3.0
Reporter: Lars George
Priority: Blocker
We seem to have an issue with store file locality as soon as a single server is
missing or faulty upon restart. HBASE-15251 is addressing a subset of the
problem, ensuring that some remaining files or an empty server do not trigger a
full recovery, but is missing further, similar cases. Especially, the case
where a server fails to report in before the active master is finished waiting
for them to do so, or where servers have been decomissioned (but not removed
from the {{regionservers}} file), and finally the true case of a dead node.
In case a single node is faulty, the user regions are _not_ assigned as saved
in the {{hbase:meta}} table, but completely randomized in a round-robin
fashion. An additional factor is that in this case the regions are _not_
assigned to the best matching node (the one with a copy of the data locally),
but to any node, leaving the locality in shambles.
What is also bad, if the {{hbase.hstore.min.locality.to.skip.major.compact}}
property is left at the default {{0.0f}}, then an older region that had no
writes since the last major compaction happened is just skipped (as expected,
usually) and locality stays bad as-is. All reads for those aged-out regions
will be network reads. But in any event, having to run a major compaction after
a restart is not good anyways.
The issue is the code in
{{AssignmentManager.processDeadServersAndRegionsInTransition()}}, which is
handed a list of dead servers. But it immediately sets the {{failover}} flag
and the code
{code}
failoverCleanupDone();
if (!failover) {
// Fresh cluster startup.
LOG.info("Clean cluster startup. Don't reassign user regions");
assignAllUserRegions(allRegions);
} else {
LOG.info("Failover! Reassign user regions");
}
{code}
is not triggering the assignment of the regions to those servers that are still
present and have all their region data local. What should happen is that only
the missing regions are reassigned, just like in the case of a server failing
while the cluster is running.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-15181) A simple implementation of date based tiered compaction
[ https://issues.apache.org/jira/browse/HBASE-15181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927057#comment-15927057 ] Dave Latham commented on HBASE-15181: - It's been awhile since I looked at opentsdb's write/read patterns, but I do think date tiered compaction would be a great fit for time series data, especially if the engine is aware. To take good advantage of it, the queries would need to set the time range on the scan itself (as opposed to purely encoding time information directly in the row/column identifiers and relying on them for time bounding). I'm not sure if opentsdb does that. > A simple implementation of date based tiered compaction > --- > > Key: HBASE-15181 > URL: https://issues.apache.org/jira/browse/HBASE-15181 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Clara Xiong >Assignee: Clara Xiong > Fix For: 2.0.0, 1.3.0, 0.98.18 > > Attachments: HBASE-15181-0.98-ADD.patch, HBASE-15181-0.98.patch, > HBASE-15181-0.98.v4.patch, HBASE-15181-98.patch, HBASE-15181-ADD.patch, > HBASE-15181-branch-1.patch, HBASE-15181-master-v1.patch, > HBASE-15181-master-v2.patch, HBASE-15181-master-v3.patch, > HBASE-15181-master-v4.patch, HBASE-15181-v1.patch, HBASE-15181-v2.patch > > > This is a simple implementation of date-based tiered compaction similar to > Cassandra's for the following benefits: > 1. Improve date-range-based scan by structuring store files in date-based > tiered layout. > 2. Reduce compaction overhead. > 3. Improve TTL efficiency. > Perfect fit for the use cases that: > 1. has mostly date-based date write and scan and a focus on the most recent > data. > 2. never or rarely deletes data. > Out-of-order writes are handled gracefully. Time range overlapping among > store files is tolerated and the performance impact is minimized. > Configuration can be set at hbase-site.xml or overriden at per-table or > per-column-famly level by hbase shell. > Design spec is at > https://docs.google.com/document/d/1_AmlNb2N8Us1xICsTeGDLKIqL6T-oHoRLZ323MG_uy8/edit?usp=sharing > Results in our production is at > https://docs.google.com/document/d/1GqRtQZMMkTEWOijZc8UCTqhACNmdxBSjtAQSYIWsmGU/edit# -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15181) A simple implementation of date based tiered compaction
[ https://issues.apache.org/jira/browse/HBASE-15181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927035#comment-15927035 ] Tianying Chang commented on HBASE-15181: [~davelatham] Thanks a lot for the information. Will look through the subtasks on HBASE-15339 also. Will give it a try to apply those patches to 1.2. My feeling is this kind of compaction algorithm should be very suitable for opentsdb use case, do you know if anyone had concrete experience on the improvement from this compaction algorithm on opentsdb? > A simple implementation of date based tiered compaction > --- > > Key: HBASE-15181 > URL: https://issues.apache.org/jira/browse/HBASE-15181 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Clara Xiong >Assignee: Clara Xiong > Fix For: 2.0.0, 1.3.0, 0.98.18 > > Attachments: HBASE-15181-0.98-ADD.patch, HBASE-15181-0.98.patch, > HBASE-15181-0.98.v4.patch, HBASE-15181-98.patch, HBASE-15181-ADD.patch, > HBASE-15181-branch-1.patch, HBASE-15181-master-v1.patch, > HBASE-15181-master-v2.patch, HBASE-15181-master-v3.patch, > HBASE-15181-master-v4.patch, HBASE-15181-v1.patch, HBASE-15181-v2.patch > > > This is a simple implementation of date-based tiered compaction similar to > Cassandra's for the following benefits: > 1. Improve date-range-based scan by structuring store files in date-based > tiered layout. > 2. Reduce compaction overhead. > 3. Improve TTL efficiency. > Perfect fit for the use cases that: > 1. has mostly date-based date write and scan and a focus on the most recent > data. > 2. never or rarely deletes data. > Out-of-order writes are handled gracefully. Time range overlapping among > store files is tolerated and the performance impact is minimized. > Configuration can be set at hbase-site.xml or overriden at per-table or > per-column-famly level by hbase shell. > Design spec is at > https://docs.google.com/document/d/1_AmlNb2N8Us1xICsTeGDLKIqL6T-oHoRLZ323MG_uy8/edit?usp=sharing > Results in our production is at > https://docs.google.com/document/d/1GqRtQZMMkTEWOijZc8UCTqhACNmdxBSjtAQSYIWsmGU/edit# -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17584) Expose ScanMetrics with ResultScanner rather than Scan
[ https://issues.apache.org/jira/browse/HBASE-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926965#comment-15926965 ] Enis Soztutar commented on HBASE-17584: --- bq. Let's open a thread in the mailing list for this? I think adding method to interface is no harm for most users as they only use the interface, not implement it... We usually allow adding methods to interfaces in minor versions. The idea is that users of interfaces are not affected, but only users extending that interface. There is no distinction in code or tooling about InterfaceAudience.Public being source/binary compatible for using or extending. > Expose ScanMetrics with ResultScanner rather than Scan > -- > > Key: HBASE-17584 > URL: https://issues.apache.org/jira/browse/HBASE-17584 > Project: HBase > Issue Type: Sub-task > Components: Client, mapreduce, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17584.patch, HBASE-17584-v1.patch > > > I think this have been discussed many times... It is a bad practice to > directly modify the Scan object passed in when calling getScanner. The reason > that we can not use a copy is we need to use the Scan object to expose scan > metrics. So we need to find another way to expose the metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17707) New More Accurate Table Skew cost function/generator
[
https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926964#comment-15926964
]
Hadoop QA commented on HBASE-17707:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 23s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 98m 28s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 140m 25s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12858910/HBASE-17707-09.patch |
| JIRA Issue | HBASE-17707 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux aa697f329dbe 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / a49bc58 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6107/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6107/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> New More Accurate Table Skew cost function/generator
>
>
> Key: HBASE-17707
> URL: https://issues.apache.org/jira/browse/HBASE-17707
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 1.2.0
> Environment: CentOS Derivative with a derivative of the 3.18.43
> kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no
[jira] [Commented] (HBASE-17584) Expose ScanMetrics with ResultScanner rather than Scan
[ https://issues.apache.org/jira/browse/HBASE-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926960#comment-15926960 ] stack commented on HBASE-17584: --- +1 on commit. On thread on dev list on compatibility, you or I to open it? This area needs a bit of cleanup. Agree most will use the Interface -- especially this one -- rather than subclass but slippery slope... > Expose ScanMetrics with ResultScanner rather than Scan > -- > > Key: HBASE-17584 > URL: https://issues.apache.org/jira/browse/HBASE-17584 > Project: HBase > Issue Type: Sub-task > Components: Client, mapreduce, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17584.patch, HBASE-17584-v1.patch > > > I think this have been discussed many times... It is a bad practice to > directly modify the Scan object passed in when calling getScanner. The reason > that we can not use a copy is we need to use the Scan object to expose scan > metrics. So we need to find another way to expose the metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17780) BoundedByteBufferPool "At capacity" messages are not actionable
[
https://issues.apache.org/jira/browse/HBASE-17780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926943#comment-15926943
]
stack commented on HBASE-17780:
---
+1
> BoundedByteBufferPool "At capacity" messages are not actionable
> ---
>
> Key: HBASE-17780
> URL: https://issues.apache.org/jira/browse/HBASE-17780
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.31
>
> Attachments: HBASE-17780-branch-1.patch, HBASE-17780.patch
>
>
> This comment in BoundedByteBufferPool talks about "At capacity ..." warnings
> from this class that may appear in logs when under load:
> {code}
> * If a ByteBuffer is bigger than the configured threshold, we will just let
> the ByteBuffer go
> * rather than add it to the pool. If more ByteBuffers than the configured
> maximum instances,
> * we will not add the passed ByteBuffer to the pool; we will just drop it
> * (we will log a WARN in this case that we are at capacity).
> {code}
> First, dropping buffers when the pool is full is obviously an expected and
> normal condition. Second, there is nothing actionable about that warning.
> Might be useful for developers, perhaps. Drop it to DEBUG.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17754) [C++] RawAsyncTable
[
https://issues.apache.org/jira/browse/HBASE-17754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926857#comment-15926857
]
Xiaobing Zhou edited comment on HBASE-17754 at 3/15/17 8:00 PM:
Thanks for the new patch.
# We should keep the unit test. client-test is like system test.
# I was referring this
{noformat}
std::pair>,
Future> Get(const hbase::Get& get);
{noformat}
Why do we need the std::pair? Can we simplify this?
# This is the sequence of call
{noformat}
template
std::shared_ptr
RawAsyncTable::CreateCallerBuilder
{noformat}
returns builder, and builder->build() returns
AsyncSingleRequestRpcRetryingCaller,
then AsyncSingleRequestRpcRetryingCaller.call() returns RESP which is the
hbase::Result.
{noformat}
# you may need to do cast like this, std::shared_ptr itself is agnostic of
polymorphism of the enclosed type.
115 std::shared_ptr RpcClient::CreateRpcChannel(const
std::string ,
116 uint16_t port, std::shared_ptr ticket, int rpc_timeout) {
117
118 std::shared_ptr channel = std::make_shared<
119 RpcChannelImplementation>(shared_from_this(), host, port, ticket,
120 rpc_timeout);
121
122 /* static_pointer_cast is safe since RpcChannelImplementation derives
123* from RpcChannel, otherwise, dynamic_pointer_cast should be used. */
124 return std::static_pointer_cast(channel);
125 }
Otherwise, v2 looks fine.
was (Author: xiaobingo):
Thanks for the new patch.
# We should keep the unit test. client-test is like system test.
# I was referring this
{noformat}
std::pair>,
Future> Get(const hbase::Get& get);
{noformat}
Why do we need the std::pair? Can we simplify this?
# This is the sequence of call
{noformat}
template
std::shared_ptr
RawAsyncTable::CreateCallerBuilder
{noformat}
returns builder, and builder->build() returns
AsyncSingleRequestRpcRetryingCaller,
then AsyncSingleRequestRpcRetryingCaller.call() returns RESP which is the
hbase::Result.
{noformat}
# you may need to do cast like this, std::shared_ptr itself is agnostic of
polymorphism of the enclosed type.
115 std::shared_ptr RpcClient::CreateRpcChannel(const
std::string ,
116 uint16_t port, std::shared_ptr ticket, int rpc_timeout) {
117
118 std::shared_ptr channel = std::make_shared<
119 RpcChannelImplementation>(shared_from_this(), host, port, ticket,
120 rpc_timeout);
121
122 /* static_pointer_cast is safe since RpcChannelImplementation derives
123* from RpcChannel, otherwise, dynamic_pointer_cast should be used. */
124 return std::static_pointer_cast(channel);
125 }
Otherwise, v2 looks fine.
> [C++] RawAsyncTable
> ---
>
> Key: HBASE-17754
> URL: https://issues.apache.org/jira/browse/HBASE-17754
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: hbase-17754_v0.patch, hbase-17754_v1.patch,
> hbase-17754-v2.patch
>
>
> We need RawAsyncTable to connect {{table.h}} to the async retrying callers.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17754) [C++] RawAsyncTable
[
https://issues.apache.org/jira/browse/HBASE-17754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926857#comment-15926857
]
Xiaobing Zhou commented on HBASE-17754:
---
Thanks for the new patch.
# We should keep the unit test. client-test is like system test.
# I was referring this
{noformat}
std::pair>,
Future> Get(const hbase::Get& get);
{noformat}
Why do we need the std::pair? Can we simplify this?
# This is the sequence of call
{noformat}
template
std::shared_ptr
RawAsyncTable::CreateCallerBuilder
{noformat}
returns builder, and builder->build() returns
AsyncSingleRequestRpcRetryingCaller,
then AsyncSingleRequestRpcRetryingCaller.call() returns RESP which is the
hbase::Result.
{noformat}
# you may need to do cast like this, std::shared_ptr itself is agnostic of
polymorphism of the enclosed type.
115 std::shared_ptr RpcClient::CreateRpcChannel(const
std::string ,
116 uint16_t port, std::shared_ptr ticket, int rpc_timeout) {
117
118 std::shared_ptr channel = std::make_shared<
119 RpcChannelImplementation>(shared_from_this(), host, port, ticket,
120 rpc_timeout);
121
122 /* static_pointer_cast is safe since RpcChannelImplementation derives
123* from RpcChannel, otherwise, dynamic_pointer_cast should be used. */
124 return std::static_pointer_cast(channel);
125 }
Otherwise, v2 looks fine.
> [C++] RawAsyncTable
> ---
>
> Key: HBASE-17754
> URL: https://issues.apache.org/jira/browse/HBASE-17754
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: hbase-17754_v0.patch, hbase-17754_v1.patch,
> hbase-17754-v2.patch
>
>
> We need RawAsyncTable to connect {{table.h}} to the async retrying callers.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17778) Remove the testing code in the AsyncRequestFutureImpl
[ https://issues.apache.org/jira/browse/HBASE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926795#comment-15926795 ] Chia-Ping Tsai commented on HBASE-17778: going to commit patch on Friday if no objection > Remove the testing code in the AsyncRequestFutureImpl > - > > Key: HBASE-17778 > URL: https://issues.apache.org/jira/browse/HBASE-17778 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0, 1.4.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Trivial > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17778.branch-1.v0.patch, > HBASE-17778.branch-1.v1.patch, HBASE-17778.v0.patch, HBASE-17778.v1.patch > > > HBASE-16224 left some testing code in the AsyncRequestFutureImpl. It iterates > all mutations in order to get the data size. The cost is directly > proportional to the size of batch. We should remove it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17723) ClientAsyncPrefetchScanner may end prematurely when the size of the cache is one
[ https://issues.apache.org/jira/browse/HBASE-17723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17723: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the reviews [~yuzhih...@gmail.com] > ClientAsyncPrefetchScanner may end prematurely when the size of the cache is > one > > > Key: HBASE-17723 > URL: https://issues.apache.org/jira/browse/HBASE-17723 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 > Environment: Ubuntu 14.04 LTS / x86_64 / 16GB RAM >Reporter: Anup Halarnkar >Assignee: Chia-Ping Tsai > Fix For: 2.0.0 > > Attachments: HBASE-17723.v0.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch, HBASE-17723.v1.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch > > > Command to build and test: > mvn clean install -X > JAVA_TOOL_OPTIONS: -Xmx4G -XX:MaxPermSize=4G -Xms1G > Failure in tests: > Results : > Failed tests: > > TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync:161->testRowsSeenMetric:174->testRowsSeenMetric:204->testMetric:330 > Metric: ROWS_SCANNED Expected: 9 Actual: 8 expected:<9> but was:<8> > > TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing:78->testReplayWorksWithMemoryCompactionPolicy:149->verifyAllEditsMadeItIn:195 > Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > Tests run: 1761, Failures: 2, Errors: 0, Skipped: 6 > > This is the output for 1st test failure: > --- > Test set: org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > --- > Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.971 sec <<< > FAILURE! - in org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > testRowsSeenMetricWithAsync(org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide) > Time elapsed: 1.134 sec <<< FAILURE! > java.lang.AssertionError: Metric: ROWS_SCANNED Expected: 9 Actual: 8 > expected:<9> but was:<8> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testMetric(TestServerSideScanMetricsFromClientSide.java:330) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:204) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:174) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync(TestServerSideScanMetricsFromClientSide.java:161) > > This is output of 2nd test failure: > --- > Test set: org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > --- > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 11.911 sec > <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > testReplayWorksThoughLotsOfFlushing(org.apache.hadoop.hbase.regionserver.TestRecoveredEdits) > Time elapsed: 11.871 sec <<< FAILURE! > java.lang.AssertionError: Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.verifyAllEditsMadeItIn(TestRecoveredEdits.java:195) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksWithMemoryCompactionPolicy(TestRecoveredEdits.java:149) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing(TestRecoveredEdits.java:78) > Please let me know if you need more information > Thanks, > Anup -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17453) add Ping into HBase server for deprecated GetProtocolVersion
[ https://issues.apache.org/jira/browse/HBASE-17453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianying Chang updated HBASE-17453: --- Status: Patch Available (was: In Progress) > add Ping into HBase server for deprecated GetProtocolVersion > > > Key: HBASE-17453 > URL: https://issues.apache.org/jira/browse/HBASE-17453 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 1.2.2 >Reporter: Tianying Chang >Assignee: Tianying Chang >Priority: Minor > Fix For: 2.0.0, 1.2.5 > > Attachments: HBASE-17453-1.2.patch, HBASE-17453-master.patch, > HBASE-17453-master-v1.patch, HBASE-17453-master-v2.patch > > > Our HBase service is hosted in AWS. We saw cases where the connection between > the client (Asynchbase in our case) and server stop working but did not throw > any exception, therefore traffic stuck. So we added a "Ping" feature in > AsyncHBase 1.5 by utilizing the GetProtocolVersion() API provided at RS side, > if no traffic for given time, we send the "Ping", if no response back for > "Ping", we assume the connect is bad and reconnect. > Now we are upgrading cluster from 94 to 1.2. However, GetProtocolVersion() is > deprecated. To be able to support same detect/reconnect feature, we added > Ping() in our internal HBase 1.2 branch, and also patched accordingly in > Asynchbase 1.7. > We would like to open source this feature since it is useful for use case in > AWS environment. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17453) add Ping into HBase server for deprecated GetProtocolVersion
[ https://issues.apache.org/jira/browse/HBASE-17453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianying Chang updated HBASE-17453: --- Status: In Progress (was: Patch Available) > add Ping into HBase server for deprecated GetProtocolVersion > > > Key: HBASE-17453 > URL: https://issues.apache.org/jira/browse/HBASE-17453 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 1.2.2 >Reporter: Tianying Chang >Assignee: Tianying Chang >Priority: Minor > Fix For: 2.0.0, 1.2.5 > > Attachments: HBASE-17453-1.2.patch, HBASE-17453-master.patch, > HBASE-17453-master-v1.patch, HBASE-17453-master-v2.patch > > > Our HBase service is hosted in AWS. We saw cases where the connection between > the client (Asynchbase in our case) and server stop working but did not throw > any exception, therefore traffic stuck. So we added a "Ping" feature in > AsyncHBase 1.5 by utilizing the GetProtocolVersion() API provided at RS side, > if no traffic for given time, we send the "Ping", if no response back for > "Ping", we assume the connect is bad and reconnect. > Now we are upgrading cluster from 94 to 1.2. However, GetProtocolVersion() is > deprecated. To be able to support same detect/reconnect feature, we added > Ping() in our internal HBase 1.2 branch, and also patched accordingly in > Asynchbase 1.7. > We would like to open source this feature since it is useful for use case in > AWS environment. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17453) add Ping into HBase server for deprecated GetProtocolVersion
[ https://issues.apache.org/jira/browse/HBASE-17453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianying Chang updated HBASE-17453: --- Attachment: HBASE-17453-master-v2.patch > add Ping into HBase server for deprecated GetProtocolVersion > > > Key: HBASE-17453 > URL: https://issues.apache.org/jira/browse/HBASE-17453 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 1.2.2 >Reporter: Tianying Chang >Assignee: Tianying Chang >Priority: Minor > Fix For: 2.0.0, 1.2.5 > > Attachments: HBASE-17453-1.2.patch, HBASE-17453-master.patch, > HBASE-17453-master-v1.patch, HBASE-17453-master-v2.patch > > > Our HBase service is hosted in AWS. We saw cases where the connection between > the client (Asynchbase in our case) and server stop working but did not throw > any exception, therefore traffic stuck. So we added a "Ping" feature in > AsyncHBase 1.5 by utilizing the GetProtocolVersion() API provided at RS side, > if no traffic for given time, we send the "Ping", if no response back for > "Ping", we assume the connect is bad and reconnect. > Now we are upgrading cluster from 94 to 1.2. However, GetProtocolVersion() is > deprecated. To be able to support same detect/reconnect feature, we added > Ping() in our internal HBase 1.2 branch, and also patched accordingly in > Asynchbase 1.7. > We would like to open source this feature since it is useful for use case in > AWS environment. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17707) New More Accurate Table Skew cost function/generator
[ https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926707#comment-15926707 ] Kahlil Oppenheimer commented on HBASE-17707: [~tedyu]: I just fixed the patch so that it should no longer fail that test. I diagnosed the problem as the RegionReplicaHostCostFunction would produce very small values as the cluster scales to be large. In clusters with lots of regions, this would make the balancer choose plans that put two replicas of the same region on the same host. To prevent this from happening, I square-rooted the RegionReplicaHostCostFunction to better distribute the values in the range 0-1 even as the cluster scales up in size. > New More Accurate Table Skew cost function/generator > > > Key: HBASE-17707 > URL: https://issues.apache.org/jira/browse/HBASE-17707 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Kahlil Oppenheimer >Assignee: Kahlil Oppenheimer >Priority: Minor > Fix For: 2.0 > > Attachments: HBASE-17707-00.patch, HBASE-17707-01.patch, > HBASE-17707-02.patch, HBASE-17707-03.patch, HBASE-17707-04.patch, > HBASE-17707-05.patch, HBASE-17707-06.patch, HBASE-17707-07.patch, > HBASE-17707-08.patch, HBASE-17707-09.patch, test-balancer2-13617.out > > > This patch includes new version of the TableSkewCostFunction and a new > TableSkewCandidateGenerator. > The new TableSkewCostFunction computes table skew by counting the minimal > number of region moves required for a given table to perfectly balance the > table across the cluster (i.e. as if the regions from that table had been > round-robin-ed across the cluster). This number of moves is computer for each > table, then normalized to a score between 0-1 by dividing by the number of > moves required in the absolute worst case (i.e. the entire table is stored on > one server), and stored in an array. The cost function then takes a weighted > average of the average and maximum value across all tables. The weights in > this average are configurable to allow for certain users to more strongly > penalize situations where one table is skewed versus where every table is a > little bit skewed. To better spread this value more evenly across the range > 0-1, we take the square root of the weighted average to get the final value. > The new TableSkewCandidateGenerator generates region moves/swaps to optimize > the above TableSkewCostFunction. It first simply tries to move regions until > each server has the right number of regions, then it swaps regions around > such that each region swap improves table skew across the cluster. > We tested the cost function and generator in our production clusters with > 100s of TBs of data and 100s of tables across dozens of servers and found > both to be very performant and accurate. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17707) New More Accurate Table Skew cost function/generator
[ https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kahlil Oppenheimer updated HBASE-17707: --- Status: Patch Available (was: Open) > New More Accurate Table Skew cost function/generator > > > Key: HBASE-17707 > URL: https://issues.apache.org/jira/browse/HBASE-17707 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Kahlil Oppenheimer >Assignee: Kahlil Oppenheimer >Priority: Minor > Fix For: 2.0 > > Attachments: HBASE-17707-00.patch, HBASE-17707-01.patch, > HBASE-17707-02.patch, HBASE-17707-03.patch, HBASE-17707-04.patch, > HBASE-17707-05.patch, HBASE-17707-06.patch, HBASE-17707-07.patch, > HBASE-17707-08.patch, HBASE-17707-09.patch, test-balancer2-13617.out > > > This patch includes new version of the TableSkewCostFunction and a new > TableSkewCandidateGenerator. > The new TableSkewCostFunction computes table skew by counting the minimal > number of region moves required for a given table to perfectly balance the > table across the cluster (i.e. as if the regions from that table had been > round-robin-ed across the cluster). This number of moves is computer for each > table, then normalized to a score between 0-1 by dividing by the number of > moves required in the absolute worst case (i.e. the entire table is stored on > one server), and stored in an array. The cost function then takes a weighted > average of the average and maximum value across all tables. The weights in > this average are configurable to allow for certain users to more strongly > penalize situations where one table is skewed versus where every table is a > little bit skewed. To better spread this value more evenly across the range > 0-1, we take the square root of the weighted average to get the final value. > The new TableSkewCandidateGenerator generates region moves/swaps to optimize > the above TableSkewCostFunction. It first simply tries to move regions until > each server has the right number of regions, then it swaps regions around > such that each region swap improves table skew across the cluster. > We tested the cost function and generator in our production clusters with > 100s of TBs of data and 100s of tables across dozens of servers and found > both to be very performant and accurate. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17707) New More Accurate Table Skew cost function/generator
[ https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kahlil Oppenheimer updated HBASE-17707: --- Attachment: HBASE-17707-09.patch > New More Accurate Table Skew cost function/generator > > > Key: HBASE-17707 > URL: https://issues.apache.org/jira/browse/HBASE-17707 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Kahlil Oppenheimer >Assignee: Kahlil Oppenheimer >Priority: Minor > Fix For: 2.0 > > Attachments: HBASE-17707-00.patch, HBASE-17707-01.patch, > HBASE-17707-02.patch, HBASE-17707-03.patch, HBASE-17707-04.patch, > HBASE-17707-05.patch, HBASE-17707-06.patch, HBASE-17707-07.patch, > HBASE-17707-08.patch, HBASE-17707-09.patch, test-balancer2-13617.out > > > This patch includes new version of the TableSkewCostFunction and a new > TableSkewCandidateGenerator. > The new TableSkewCostFunction computes table skew by counting the minimal > number of region moves required for a given table to perfectly balance the > table across the cluster (i.e. as if the regions from that table had been > round-robin-ed across the cluster). This number of moves is computer for each > table, then normalized to a score between 0-1 by dividing by the number of > moves required in the absolute worst case (i.e. the entire table is stored on > one server), and stored in an array. The cost function then takes a weighted > average of the average and maximum value across all tables. The weights in > this average are configurable to allow for certain users to more strongly > penalize situations where one table is skewed versus where every table is a > little bit skewed. To better spread this value more evenly across the range > 0-1, we take the square root of the weighted average to get the final value. > The new TableSkewCandidateGenerator generates region moves/swaps to optimize > the above TableSkewCostFunction. It first simply tries to move regions until > each server has the right number of regions, then it swaps regions around > such that each region swap improves table skew across the cluster. > We tested the cost function and generator in our production clusters with > 100s of TBs of data and 100s of tables across dozens of servers and found > both to be very performant and accurate. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17723) ClientAsyncPrefetchScanner may end prematurely when the size of the cache is one
[ https://issues.apache.org/jira/browse/HBASE-17723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926697#comment-15926697 ] Chia-Ping Tsai commented on HBASE-17723: bq. Make this your first commit ? I will commit it after reading the guide. > ClientAsyncPrefetchScanner may end prematurely when the size of the cache is > one > > > Key: HBASE-17723 > URL: https://issues.apache.org/jira/browse/HBASE-17723 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 > Environment: Ubuntu 14.04 LTS / x86_64 / 16GB RAM >Reporter: Anup Halarnkar >Assignee: Chia-Ping Tsai > Fix For: 2.0.0 > > Attachments: HBASE-17723.v0.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch, HBASE-17723.v1.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch > > > Command to build and test: > mvn clean install -X > JAVA_TOOL_OPTIONS: -Xmx4G -XX:MaxPermSize=4G -Xms1G > Failure in tests: > Results : > Failed tests: > > TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync:161->testRowsSeenMetric:174->testRowsSeenMetric:204->testMetric:330 > Metric: ROWS_SCANNED Expected: 9 Actual: 8 expected:<9> but was:<8> > > TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing:78->testReplayWorksWithMemoryCompactionPolicy:149->verifyAllEditsMadeItIn:195 > Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > Tests run: 1761, Failures: 2, Errors: 0, Skipped: 6 > > This is the output for 1st test failure: > --- > Test set: org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > --- > Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.971 sec <<< > FAILURE! - in org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > testRowsSeenMetricWithAsync(org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide) > Time elapsed: 1.134 sec <<< FAILURE! > java.lang.AssertionError: Metric: ROWS_SCANNED Expected: 9 Actual: 8 > expected:<9> but was:<8> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testMetric(TestServerSideScanMetricsFromClientSide.java:330) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:204) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:174) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync(TestServerSideScanMetricsFromClientSide.java:161) > > This is output of 2nd test failure: > --- > Test set: org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > --- > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 11.911 sec > <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > testReplayWorksThoughLotsOfFlushing(org.apache.hadoop.hbase.regionserver.TestRecoveredEdits) > Time elapsed: 11.871 sec <<< FAILURE! > java.lang.AssertionError: Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.verifyAllEditsMadeItIn(TestRecoveredEdits.java:195) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksWithMemoryCompactionPolicy(TestRecoveredEdits.java:149) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing(TestRecoveredEdits.java:78) > Please let me know if you need more information > Thanks, > Anup -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17723) ClientAsyncPrefetchScanner may end prematurely when the size of the cache is one
[ https://issues.apache.org/jira/browse/HBASE-17723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926668#comment-15926668 ] Ted Yu commented on HBASE-17723: I see. Make this your first commit ? > ClientAsyncPrefetchScanner may end prematurely when the size of the cache is > one > > > Key: HBASE-17723 > URL: https://issues.apache.org/jira/browse/HBASE-17723 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 > Environment: Ubuntu 14.04 LTS / x86_64 / 16GB RAM >Reporter: Anup Halarnkar >Assignee: Chia-Ping Tsai > Fix For: 2.0.0 > > Attachments: HBASE-17723.v0.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch, HBASE-17723.v1.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch > > > Command to build and test: > mvn clean install -X > JAVA_TOOL_OPTIONS: -Xmx4G -XX:MaxPermSize=4G -Xms1G > Failure in tests: > Results : > Failed tests: > > TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync:161->testRowsSeenMetric:174->testRowsSeenMetric:204->testMetric:330 > Metric: ROWS_SCANNED Expected: 9 Actual: 8 expected:<9> but was:<8> > > TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing:78->testReplayWorksWithMemoryCompactionPolicy:149->verifyAllEditsMadeItIn:195 > Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > Tests run: 1761, Failures: 2, Errors: 0, Skipped: 6 > > This is the output for 1st test failure: > --- > Test set: org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > --- > Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.971 sec <<< > FAILURE! - in org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > testRowsSeenMetricWithAsync(org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide) > Time elapsed: 1.134 sec <<< FAILURE! > java.lang.AssertionError: Metric: ROWS_SCANNED Expected: 9 Actual: 8 > expected:<9> but was:<8> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testMetric(TestServerSideScanMetricsFromClientSide.java:330) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:204) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:174) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync(TestServerSideScanMetricsFromClientSide.java:161) > > This is output of 2nd test failure: > --- > Test set: org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > --- > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 11.911 sec > <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > testReplayWorksThoughLotsOfFlushing(org.apache.hadoop.hbase.regionserver.TestRecoveredEdits) > Time elapsed: 11.871 sec <<< FAILURE! > java.lang.AssertionError: Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.verifyAllEditsMadeItIn(TestRecoveredEdits.java:195) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksWithMemoryCompactionPolicy(TestRecoveredEdits.java:149) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing(TestRecoveredEdits.java:78) > Please let me know if you need more information > Thanks, > Anup -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17723) ClientAsyncPrefetchScanner may end prematurely when the size of the cache is one
[ https://issues.apache.org/jira/browse/HBASE-17723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926664#comment-15926664 ] Chia-Ping Tsai commented on HBASE-17723: [~yuzhih...@gmail.com] The HBASE-17781 is unrelated with this jira. # i run the TestAcidGuarantees 10 times. All pass. # The TestAcidGuarantees doesn't call async scan > ClientAsyncPrefetchScanner may end prematurely when the size of the cache is > one > > > Key: HBASE-17723 > URL: https://issues.apache.org/jira/browse/HBASE-17723 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 > Environment: Ubuntu 14.04 LTS / x86_64 / 16GB RAM >Reporter: Anup Halarnkar >Assignee: Chia-Ping Tsai > Fix For: 2.0.0 > > Attachments: HBASE-17723.v0.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch, HBASE-17723.v1.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch > > > Command to build and test: > mvn clean install -X > JAVA_TOOL_OPTIONS: -Xmx4G -XX:MaxPermSize=4G -Xms1G > Failure in tests: > Results : > Failed tests: > > TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync:161->testRowsSeenMetric:174->testRowsSeenMetric:204->testMetric:330 > Metric: ROWS_SCANNED Expected: 9 Actual: 8 expected:<9> but was:<8> > > TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing:78->testReplayWorksWithMemoryCompactionPolicy:149->verifyAllEditsMadeItIn:195 > Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > Tests run: 1761, Failures: 2, Errors: 0, Skipped: 6 > > This is the output for 1st test failure: > --- > Test set: org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > --- > Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.971 sec <<< > FAILURE! - in org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > testRowsSeenMetricWithAsync(org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide) > Time elapsed: 1.134 sec <<< FAILURE! > java.lang.AssertionError: Metric: ROWS_SCANNED Expected: 9 Actual: 8 > expected:<9> but was:<8> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testMetric(TestServerSideScanMetricsFromClientSide.java:330) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:204) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:174) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync(TestServerSideScanMetricsFromClientSide.java:161) > > This is output of 2nd test failure: > --- > Test set: org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > --- > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 11.911 sec > <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > testReplayWorksThoughLotsOfFlushing(org.apache.hadoop.hbase.regionserver.TestRecoveredEdits) > Time elapsed: 11.871 sec <<< FAILURE! > java.lang.AssertionError: Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.verifyAllEditsMadeItIn(TestRecoveredEdits.java:195) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksWithMemoryCompactionPolicy(TestRecoveredEdits.java:149) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing(TestRecoveredEdits.java:78) > Please let me know if you need more information > Thanks, > Anup -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17785) RSGroupBasedLoadBalancer fails to assign new table regions when cloning snapshot
[
https://issues.apache.org/jira/browse/HBASE-17785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925083#comment-15925083
]
Andrew Purtell edited comment on HBASE-17785 at 3/15/17 5:52 PM:
-
Patch with a new test.
The equivalent for a branch-1 backport would be different. I'm carrying that on
an internal dev branch. Not sure where to put it.
FWIW, for branch-1 implementing coprocessor interfaces is done a bit
differently, because we are not using default methods there. New unit test case
is the same.
{code}
diff --git
a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
b/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
index 875ba17c9a..420eb45443 100644
---
a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
+++
b/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
@@ -680,7 +680,7 @@ public class RSGroupAdminEndpoint extends
RSGroupAdminService
public void preCloneSnapshot(ObserverContext
ctx,
SnapshotDescription snapshot,
HTableDescriptor hTableDescriptor) throws
IOException {
-
+groupAdminServer.prepareRSGroupForTable(hTableDescriptor);
}
@Override
{code}
was (Author: apurtell):
Patch with a new test.
The equivalent for a branch-1 backport would be different. I'm carrying that on
an internal dev branch. Not sure where to put it.
> RSGroupBasedLoadBalancer fails to assign new table regions when cloning
> snapshot
>
>
> Key: HBASE-17785
> URL: https://issues.apache.org/jira/browse/HBASE-17785
> Project: HBase
> Issue Type: Bug
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Fix For: 2.0.0
>
> Attachments: HBASE-17785.patch
>
>
> A novice starting out with RSGroupBasedLoadBalancer will want to enable it
> and, before assigning tables to groups, may want to create some test tables.
> Currently that does not work when creating a table by cloning a snapshot, in
> a surprising way. All regions of the table fail to open yet it is moved into
> ENABLED state. The client hangs indefinitely.
> {noformat}
> 2017-03-14 19:25:49,833 INFO [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> snapshot.CloneSnapshotHandler: Clone snapshot=seed on table=test_1 completed!
> 2017-03-14 19:25:49,871 INFO [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> hbase.MetaTableAccessor: Added 25
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN
[jira] [Commented] (HBASE-17453) add Ping into HBase server for deprecated GetProtocolVersion
[ https://issues.apache.org/jira/browse/HBASE-17453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926652#comment-15926652 ] Tianying Chang commented on HBASE-17453: [~anoop.hbase] Yes, I agree the input parameter for ping is useless. will remove that. and thanks for catching included the MasterProtos.java into the package, will remove that also. [~saint@gmail.com] Our usage is to call from asynchbase client to test the client/server connection, so I feel it is better to be in Client.proto. But I am fine to put it in Admin.proto if it can achieve the same goal. As to coprocessor, that means all the tables in the cluster has to deploy the noop coprocessor, right? If so, that seems too much operation overhead. > add Ping into HBase server for deprecated GetProtocolVersion > > > Key: HBASE-17453 > URL: https://issues.apache.org/jira/browse/HBASE-17453 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 1.2.2 >Reporter: Tianying Chang >Assignee: Tianying Chang >Priority: Minor > Fix For: 2.0.0, 1.2.5 > > Attachments: HBASE-17453-1.2.patch, HBASE-17453-master.patch, > HBASE-17453-master-v1.patch > > > Our HBase service is hosted in AWS. We saw cases where the connection between > the client (Asynchbase in our case) and server stop working but did not throw > any exception, therefore traffic stuck. So we added a "Ping" feature in > AsyncHBase 1.5 by utilizing the GetProtocolVersion() API provided at RS side, > if no traffic for given time, we send the "Ping", if no response back for > "Ping", we assume the connect is bad and reconnect. > Now we are upgrading cluster from 94 to 1.2. However, GetProtocolVersion() is > deprecated. To be able to support same detect/reconnect feature, we added > Ping() in our internal HBase 1.2 branch, and also patched accordingly in > Asynchbase 1.7. > We would like to open source this feature since it is useful for use case in > AWS environment. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17785) RSGroupBasedLoadBalancer fails to assign new table regions when cloning snapshot
[
https://issues.apache.org/jira/browse/HBASE-17785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926640#comment-15926640
]
Andrew Purtell commented on HBASE-17785:
[~saint@gmail.com] [~toffer] WDYT?
> RSGroupBasedLoadBalancer fails to assign new table regions when cloning
> snapshot
>
>
> Key: HBASE-17785
> URL: https://issues.apache.org/jira/browse/HBASE-17785
> Project: HBase
> Issue Type: Bug
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Fix For: 2.0.0
>
> Attachments: HBASE-17785.patch
>
>
> A novice starting out with RSGroupBasedLoadBalancer will want to enable it
> and, before assigning tables to groups, may want to create some test tables.
> Currently that does not work when creating a table by cloning a snapshot, in
> a surprising way. All regions of the table fail to open yet it is moved into
> ENABLED state. The client hangs indefinitely.
> {noformat}
> 2017-03-14 19:25:49,833 INFO [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> snapshot.CloneSnapshotHandler: Clone snapshot=seed on table=test_1 completed!
> 2017-03-14 19:25:49,871 INFO [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> hbase.MetaTableAccessor: Added 25
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14 19:25:49,875 WARN [MASTER_TABLE_OPERATIONS-ip-172-31-5-95:8100-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table test_1 is null
> 2017-03-14
[jira] [Commented] (HBASE-17789) ZKConfig.getZKQuorumServersStringFromHbaseConfig creates a incorrect zkquorum string
[ https://issues.apache.org/jira/browse/HBASE-17789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926614#comment-15926614 ] Josh Elser commented on HBASE-17789: bq. a value of ""hbase.zookeeper.quorum" from list peers with value as "a:2181,b:2181c:2181:/hbase" [~vishk], this doesn't look like a valid value. HBase does not support providing a znode in this property. See https://hbase.apache.org/book.html#hbase.zookeeper.quorum An exception being thrown from that method seems logical. > ZKConfig.getZKQuorumServersStringFromHbaseConfig creates a incorrect zkquorum > string > > > Key: HBASE-17789 > URL: https://issues.apache.org/jira/browse/HBASE-17789 > Project: HBase > Issue Type: Bug >Reporter: Vishal Khandelwal >Assignee: Vishal Khandelwal > > , if we try to a hbase using "HBaseAdmin" by reading a value of > ""hbase.zookeeper.quorum" from list peers with value as > "a:2181,b:2181c:2181:/hbase". > org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersStringFromHbaseConfig > returns a incorrect queryString which causes following exception Because > last host would see :/hbase also in the string. This method was existing > earlier but never being called before above change -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15160) Put back HFile's HDFS op latency sampling code and add metrics for monitoring
[ https://issues.apache.org/jira/browse/HBASE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926523#comment-15926523 ] Yu Li commented on HBASE-15160: --- [~enis] Is it possible for us to get back on this and move on sir? In some [email thread|http://mail-archives.apache.org/mod_mbox/hbase-dev/201703.mbox/%3CCAM7-19%2BSVg%2BTWg13UO3%3DTQz3KQSoVu3FkaBLdR%3DdEb2fRimCKw%40mail.gmail.com%3E] user is asking when this could get committed. Thanks. > Put back HFile's HDFS op latency sampling code and add metrics for monitoring > - > > Key: HBASE-15160 > URL: https://issues.apache.org/jira/browse/HBASE-15160 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0, 1.1.2 >Reporter: Yu Li >Assignee: Yu Li > Attachments: HBASE-15160.patch, HBASE-15160_v2.patch, > HBASE-15160_v3.patch, hbase-15160_v4.patch, hbase-15160_v5.patch > > > In HBASE-11586 all HDFS op latency sampling code, including fsReadLatency, > fsPreadLatency and fsWriteLatency, have been removed. There was some > discussion about putting them back in a new JIRA but never happened. > According to our experience, these metrics are useful to judge whether issue > lies on HDFS when slow request occurs, so we propose to put them back in this > JIRA, and add the metrics for monitoring as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line
[ https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926502#comment-15926502 ] Karan Mehta commented on HBASE-14925: - Hey [~stack], I have implemented this method as per Ronan's suggestions and submitted the patch. Can you please have a look? Thanks! > Develop HBase shell command/tool to list table's region info through command > line > - > > Key: HBASE-14925 > URL: https://issues.apache.org/jira/browse/HBASE-14925 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Romil Choksi >Assignee: Karan Mehta > Attachments: HBASE-14925.patch > > > I am going through the hbase shell commands to see if there is anything I can > use to get all the regions info just for a particular table. I don’t see any > such command that provides me that information. > It would be better to have a command that provides region info, start key, > end key etc taking a table name as the input parameter. This is available > through HBase UI on clicking on a particular table's link > A tool/shell command to get a list of regions for a table or all tables in a > tabular structured output (that is machine readable) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926497#comment-15926497 ] Anastasia Braginsky commented on HBASE-16438: - The CAS can work similarly to how it works today in MemStoreLABImpl. One thread wins the race buy being able to swap (CAS) the null pointer to the pointer to chunk (not yet initialized). While the chunk is being initialized the other threads may be busy-waiting. But I have a different suggestion. Initially, I actually thought about implementing myself the MemStoreAllocator (that you are calling ChunkCreator). So may be you can drop the ChunkCreator from this patch and just commit all the Cell related refactoring? I'll continue with the Allocator/Creator and all the rest... What do you think? > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-16438_1.patch, HBASE-16438.patch, > MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17698) ReplicationEndpoint choosing sinks
[ https://issues.apache.org/jira/browse/HBASE-17698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926492#comment-15926492 ] Karan Mehta commented on HBASE-17698: - Hey [~apurtell], I have made the changes as suggested. Can you have a look? Thanks! > ReplicationEndpoint choosing sinks > -- > > Key: HBASE-17698 > URL: https://issues.apache.org/jira/browse/HBASE-17698 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0 >Reporter: churro morales >Assignee: Karan Mehta > Attachments: HBASE-17698.patch > > > The only time we choose new sinks is when we have a ConnectException, but we > have encountered other exceptions where there is a problem contacting a > particular sink and replication gets backed up for any sources that try that > sink > HBASE-17675 occurred when there was a bad keytab refresh and the source was > stuck. > Another issue we recently had was a bad drive controller on the sink side and > replication was stuck again. > Is there any reason not to choose new sinks anytime we have a > RemoteException? I can understand TableNotFound we don't have to choose new > sinks, but for all other cases this seems like the safest approach. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16584) Backport the new ipc implementation in HBASE-16432 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926486#comment-15926486 ] Yu Li commented on HBASE-16584: --- Left some comments in RB, mainly about javadoc, FYI. [~Apache9] > Backport the new ipc implementation in HBASE-16432 to branch-1 > -- > > Key: HBASE-16584 > URL: https://issues.apache.org/jira/browse/HBASE-16584 > Project: HBase > Issue Type: Task > Components: Client, IPC/RPC >Affects Versions: 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 1.4.0 > > Attachments: HBASE-16584-branch-1.patch, > HBASE-16584-branch-1-v1.patch, HBASE-16584-branch-1-v2.patch, > HBASE-16584-branch-1-v3.patch > > > Two problems. > First, as RpcCliemtImpl is the default implementation on branch-1, we need to > confirm that the modification on master does not make it slower. I'm not sure > but I used a big lock to protect everything in the new implementation, so it > may have bad impact on performance. > Second, some configurations of the old AsyncRpcClient is gone. Such as > "hbase.rpc.client.threads.max" and "hbase.rpc.client.nativetransport". Now > You could pass a EventLoopGroup object directly through a helper class which > makes it more flexible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17723) ClientAsyncPrefetchScanner may end prematurely when the size of the cache is one
[ https://issues.apache.org/jira/browse/HBASE-17723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926428#comment-15926428 ] Chia-Ping Tsai commented on HBASE-17723: [~yuzhih...@gmail.com] i will check out it. Thanks for your reminder. > ClientAsyncPrefetchScanner may end prematurely when the size of the cache is > one > > > Key: HBASE-17723 > URL: https://issues.apache.org/jira/browse/HBASE-17723 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 > Environment: Ubuntu 14.04 LTS / x86_64 / 16GB RAM >Reporter: Anup Halarnkar >Assignee: Chia-Ping Tsai > Fix For: 2.0.0 > > Attachments: HBASE-17723.v0.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch, HBASE-17723.v1.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch > > > Command to build and test: > mvn clean install -X > JAVA_TOOL_OPTIONS: -Xmx4G -XX:MaxPermSize=4G -Xms1G > Failure in tests: > Results : > Failed tests: > > TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync:161->testRowsSeenMetric:174->testRowsSeenMetric:204->testMetric:330 > Metric: ROWS_SCANNED Expected: 9 Actual: 8 expected:<9> but was:<8> > > TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing:78->testReplayWorksWithMemoryCompactionPolicy:149->verifyAllEditsMadeItIn:195 > Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > Tests run: 1761, Failures: 2, Errors: 0, Skipped: 6 > > This is the output for 1st test failure: > --- > Test set: org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > --- > Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.971 sec <<< > FAILURE! - in org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > testRowsSeenMetricWithAsync(org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide) > Time elapsed: 1.134 sec <<< FAILURE! > java.lang.AssertionError: Metric: ROWS_SCANNED Expected: 9 Actual: 8 > expected:<9> but was:<8> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testMetric(TestServerSideScanMetricsFromClientSide.java:330) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:204) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:174) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync(TestServerSideScanMetricsFromClientSide.java:161) > > This is output of 2nd test failure: > --- > Test set: org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > --- > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 11.911 sec > <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > testReplayWorksThoughLotsOfFlushing(org.apache.hadoop.hbase.regionserver.TestRecoveredEdits) > Time elapsed: 11.871 sec <<< FAILURE! > java.lang.AssertionError: Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.verifyAllEditsMadeItIn(TestRecoveredEdits.java:195) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksWithMemoryCompactionPolicy(TestRecoveredEdits.java:149) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing(TestRecoveredEdits.java:78) > Please let me know if you need more information > Thanks, > Anup -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17584) Expose ScanMetrics with ResultScanner rather than Scan
[ https://issues.apache.org/jira/browse/HBASE-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926426#comment-15926426 ] Hudson commented on HBASE-17584: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2677 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2677/]) HBASE-17584 Expose ScanMetrics with ResultScanner rather than Scan (zhangduo: rev a49bc58a5456c2974552c7b6ffab9ea39393ca78) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/RegionAsTable.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTableResultScanner.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java * (edit) hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/client/ClientSideRegionScanner.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/TestServerSideScanMetricsFromClientSide.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractClientScanner.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ResultScanner.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/metrics/ServerSideScanMetrics.java > Expose ScanMetrics with ResultScanner rather than Scan > -- > > Key: HBASE-17584 > URL: https://issues.apache.org/jira/browse/HBASE-17584 > Project: HBase > Issue Type: Sub-task > Components: Client, mapreduce, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17584.patch, HBASE-17584-v1.patch > > > I think this have been discussed many times... It is a bad practice to > directly modify the Scan object passed in when calling getScanner. The reason > that we can not use a copy is we need to use the Scan object to expose scan > metrics. So we need to find another way to expose the metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17723) ClientAsyncPrefetchScanner may end prematurely when the size of the cache is one
[ https://issues.apache.org/jira/browse/HBASE-17723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926329#comment-15926329 ] Ted Yu commented on HBASE-17723: Have you seen HBASE-17781 ? > ClientAsyncPrefetchScanner may end prematurely when the size of the cache is > one > > > Key: HBASE-17723 > URL: https://issues.apache.org/jira/browse/HBASE-17723 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 > Environment: Ubuntu 14.04 LTS / x86_64 / 16GB RAM >Reporter: Anup Halarnkar >Assignee: Chia-Ping Tsai > Fix For: 2.0.0 > > Attachments: HBASE-17723.v0.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch, HBASE-17723.v1.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch > > > Command to build and test: > mvn clean install -X > JAVA_TOOL_OPTIONS: -Xmx4G -XX:MaxPermSize=4G -Xms1G > Failure in tests: > Results : > Failed tests: > > TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync:161->testRowsSeenMetric:174->testRowsSeenMetric:204->testMetric:330 > Metric: ROWS_SCANNED Expected: 9 Actual: 8 expected:<9> but was:<8> > > TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing:78->testReplayWorksWithMemoryCompactionPolicy:149->verifyAllEditsMadeItIn:195 > Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > Tests run: 1761, Failures: 2, Errors: 0, Skipped: 6 > > This is the output for 1st test failure: > --- > Test set: org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > --- > Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.971 sec <<< > FAILURE! - in org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > testRowsSeenMetricWithAsync(org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide) > Time elapsed: 1.134 sec <<< FAILURE! > java.lang.AssertionError: Metric: ROWS_SCANNED Expected: 9 Actual: 8 > expected:<9> but was:<8> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testMetric(TestServerSideScanMetricsFromClientSide.java:330) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:204) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:174) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync(TestServerSideScanMetricsFromClientSide.java:161) > > This is output of 2nd test failure: > --- > Test set: org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > --- > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 11.911 sec > <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > testReplayWorksThoughLotsOfFlushing(org.apache.hadoop.hbase.regionserver.TestRecoveredEdits) > Time elapsed: 11.871 sec <<< FAILURE! > java.lang.AssertionError: Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.verifyAllEditsMadeItIn(TestRecoveredEdits.java:195) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksWithMemoryCompactionPolicy(TestRecoveredEdits.java:149) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing(TestRecoveredEdits.java:78) > Please let me know if you need more information > Thanks, > Anup -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15454) Freeze date tiered store files older than max age
[ https://issues.apache.org/jira/browse/HBASE-15454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Latham updated HBASE-15454: Summary: Freeze date tiered store files older than max age (was: Archive store files older than max age) > Freeze date tiered store files older than max age > - > > Key: HBASE-15454 > URL: https://issues.apache.org/jira/browse/HBASE-15454 > Project: HBase > Issue Type: New Feature > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 0.98.18, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15454.patch, HBASE-15454-v1.patch, > HBASE-15454-v2.patch, HBASE-15454-v3.patch, HBASE-15454-v4.patch, > HBASE-15454-v5.patch, HBASE-15454-v6.patch, HBASE-15454-v7.patch > > > In date tiered compaction, the store files older than max age are never > touched by minor compactions. Here we introduce a 'freeze window' operation, > which does the follow things: > 1. Find all store files that contains cells whose timestamp are in the give > window. > 2. Compaction all these files and output one file for each window that these > files covered. > After the compaction, we will have only one in the give window, and all cells > whose timestamp are in the give window are in the only file. And if you do > not write new cells with an older timestamp in this window, the file will > never be changed. This makes it easier to do erasure coding on the freezed > file to reduce redundence. And also, it makes it possible to check > consistency between master and peer cluster incrementally. > And why use the word 'freeze'? > Because there is already an 'HFileArchiver' class. I want to use a different > word to prevent confusing. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15368) Add pluggable window support
[
https://issues.apache.org/jira/browse/HBASE-15368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Latham updated HBASE-15368:
Fix Version/s: (was: 1.4.0)
> Add pluggable window support
>
>
> Key: HBASE-15368
> URL: https://issues.apache.org/jira/browse/HBASE-15368
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
>Affects Versions: 2.0.0, 1.3.0, 0.98.19, 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 0.98.19
>
> Attachments: HBASE-15368-0.98.patch, HBASE-15368-branch-1.patch,
> HBASE-15368.patch, HBASE-15368-v1.patch, HBASE-15368-v2.patch,
> HBASE-15368-v3.patch, HBASE-15368-v4.patch, HBASE-15368-v5.patch,
> HBASE-15368-v6.patch, HBASE-15368-v7.patch
>
>
> Abstract a {{CompactionWindow}} and a {{CompactionWindowFactory}}. Both are
> orthogonal to the implementation of DateTieredCompactionPolicy. Users can
> implement their own {{CompactionWindow}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (HBASE-15339) Improve DateTieredCompactionPolicy
[ https://issues.apache.org/jira/browse/HBASE-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Latham resolved HBASE-15339. - Resolution: Fixed > Improve DateTieredCompactionPolicy > -- > > Key: HBASE-15339 > URL: https://issues.apache.org/jira/browse/HBASE-15339 > Project: HBase > Issue Type: Improvement > Components: Compaction >Reporter: Duo Zhang > Fix For: 2.0.0, 0.98.19, 1.3.0 > > > Add multi-output support. > Add archive old data support. > ... -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15339) Improve DateTieredCompactionPolicy
[ https://issues.apache.org/jira/browse/HBASE-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Latham updated HBASE-15339: Fix Version/s: 1.3.0 0.98.19 2.0.0 > Improve DateTieredCompactionPolicy > -- > > Key: HBASE-15339 > URL: https://issues.apache.org/jira/browse/HBASE-15339 > Project: HBase > Issue Type: Improvement > Components: Compaction >Reporter: Duo Zhang > Fix For: 2.0.0, 1.3.0, 0.98.19 > > > Add multi-output support. > Add archive old data support. > ... -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15389) Write out multiple files when compaction
[
https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Latham updated HBASE-15389:
Fix Version/s: (was: 1.4.0)
> Write out multiple files when compaction
>
>
> Key: HBASE-15389
> URL: https://issues.apache.org/jira/browse/HBASE-15389
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
>Affects Versions: 2.0.0, 0.98.19, 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 0.98.19
>
> Attachments: HBASE-15389-0.98.patch, HBASE-15389-0.98.v1.patch,
> HBASE-15389-0.98.v2.patch, HBASE-15389-0.98.v3.patch,
> HBASE-15389-branch-1.patch, HBASE-15389-branch-1.v1.patch, HBASE-15389.patch,
> HBASE-15389-uc.patch, HBASE-15389-v10.patch, HBASE-15389-v11.patch,
> HBASE-15389-v12.patch, HBASE-15389-v13.patch, HBASE-15389-v14.patch,
> HBASE-15389-v1.patch, HBASE-15389-v2.patch, HBASE-15389-v3.patch,
> HBASE-15389-v4.patch, HBASE-15389-v5.patch, HBASE-15389-v6.patch,
> HBASE-15389-v7.patch, HBASE-15389-v8.patch, HBASE-15389-v9.patch
>
>
> Add {{DateTieredCompactor}} and {{DateTieredMultiFileWriter}} to support
> writing out multiple files based on timestamp windows when doing date tiered
> compactions.
> Abstract {{AbstractMultiOutputCompactor}} and {{AbstractMultiFileWriter}}
> which contain the general logic for multi output and can be used to implement
> new compaction policies that requires multi output in the future.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-15527) Refactor Compactor related classes
[ https://issues.apache.org/jira/browse/HBASE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Latham updated HBASE-15527: Fix Version/s: (was: 1.4.0) > Refactor Compactor related classes > -- > > Key: HBASE-15527 > URL: https://issues.apache.org/jira/browse/HBASE-15527 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 0.98.18, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.3.0, 0.98.19 > > Attachments: HBASE-15527-0.98.patch, HBASE-15527-branch-1.patch, > HBASE-15527-branch-1.v1.patch, HBASE-15527.patch, HBASE-15527-v1.patch > > > DefaultCompactor and MultiOutputCompactor still have a lot of duplicate code > now, need refactoring. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15665) Support using different StoreFileComparators for different CompactionPolicies
[ https://issues.apache.org/jira/browse/HBASE-15665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Latham updated HBASE-15665: Fix Version/s: (was: 1.4.0) > Support using different StoreFileComparators for different CompactionPolicies > - > > Key: HBASE-15665 > URL: https://issues.apache.org/jira/browse/HBASE-15665 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 0.98.19, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.3.0, 0.98.19 > > Attachments: HBASE-15665-0.98.patch, HBASE-15665-branch-1.patch, > HBASE-15665.patch > > > Now for DateTieredCompactionPolicy, if we output multiple files when > compaction, we will use the same max sequence id for all the output files(see > HBASE-15389 for more details). So we need to sort the store files by > timestamp if they have the same max sequence id which is different from the > original store file comparator. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15400) Use DateTieredCompactor for Date Tiered Compaction
[ https://issues.apache.org/jira/browse/HBASE-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Latham updated HBASE-15400: Fix Version/s: (was: 1.4.0) > Use DateTieredCompactor for Date Tiered Compaction > -- > > Key: HBASE-15400 > URL: https://issues.apache.org/jira/browse/HBASE-15400 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Clara Xiong >Assignee: Clara Xiong > Fix For: 2.0.0, 1.3.0, 0.98.19 > > Attachments: HBASE-15400-0.98.patch, HBASE-15400-15389-v12.patch, > HBASE-15400-branch-1.patch, HBASE-15400.patch, HBASE-15400-v1.pa, > HBASE-15400-v3.patch, HBASE-15400-v3-v3.patch, HBASE-15400-v3-v4.patch, > HBASE-15400-v3-v5.patch, HBASE-15400-v6.patch, HBASE-15400-v7.patch > > > When we compact, we can output multiple files along the current window > boundaries. There are two use cases: > 1. Major compaction: We want to output date tiered store files with data > older than max age archived in trunks of the window size on the higher tier. > Once a window is old enough, we don't combine the windows to promote to the > next tier any further. So files in these windows retain the same timespan as > they were minor-compacted last time, which is the window size of the highest > tier. Major compaction will touch these files and we want to maintain the > same layout. This way, TTL and archiving will be simpler and more efficient. > 2. Bulk load files and the old file generated by major compaction before > upgrading to DTCP. > Pros: > 1. Restore locality, process versioning, updates and deletes while > maintaining the tiered layout. > 2. The best way to fix a skewed layout. > > This work is based on a prototype of DateTieredCompactor from HBASE-15389 and > focused on the part to meet needs for these two use cases while supporting > others. I have to call out a few design decisions: > 1. We only want to output the files along all windows for major compaction. > And we want to output multiple files older than max age in the trunks of the > maximum tier window size determined by base window size, windows per tier and > max age. > 2. For minor compaction, we don't want to output too many files, which will > remain around because of current restriction of contiguous compaction by seq > id. I will only output two files if all the files in the windows are being > combined, one for the data within window and the other for the out-of-window > tail. If there is any file in the window excluded from compaction, only one > file will be output from compaction. When the windows are promoted, the > situation of out of order data will gradually improve. For the incoming > window, we need to accommodate the case with user-specified future data. > 3. We have to pass the boundaries with the list of store file as a complete > time snapshot instead of two separate calls because window layout is > determined by the time the computation is called. So we will need new type of > compaction request. > 4. Since we will assign the same seq id for all output files, we need to sort > by maxTimestamp subsequently. Right now all compaction policy gets the files > sorted for StoreFileManager which sorts by seq id and other criteria. I will > use this order for DTCP only, to avoid impacting other compaction policies. > 5. We need some cleanup of current design of StoreEngine and CompactionPolicy. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15454) Archive store files older than max age
[ https://issues.apache.org/jira/browse/HBASE-15454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Latham updated HBASE-15454: Issue Type: New Feature (was: Sub-task) Parent: (was: HBASE-15339) > Archive store files older than max age > -- > > Key: HBASE-15454 > URL: https://issues.apache.org/jira/browse/HBASE-15454 > Project: HBase > Issue Type: New Feature > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 0.98.18, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15454.patch, HBASE-15454-v1.patch, > HBASE-15454-v2.patch, HBASE-15454-v3.patch, HBASE-15454-v4.patch, > HBASE-15454-v5.patch, HBASE-15454-v6.patch, HBASE-15454-v7.patch > > > In date tiered compaction, the store files older than max age are never > touched by minor compactions. Here we introduce a 'freeze window' operation, > which does the follow things: > 1. Find all store files that contains cells whose timestamp are in the give > window. > 2. Compaction all these files and output one file for each window that these > files covered. > After the compaction, we will have only one in the give window, and all cells > whose timestamp are in the give window are in the only file. And if you do > not write new cells with an older timestamp in this window, the file will > never be changed. This makes it easier to do erasure coding on the freezed > file to reduce redundence. And also, it makes it possible to check > consistency between master and peer cluster incrementally. > And why use the word 'freeze'? > Because there is already an 'HFileArchiver' class. I want to use a different > word to prevent confusing. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17739) BucketCache is inefficient/wasteful/dumb in its bucket allocations
[ https://issues.apache.org/jira/browse/HBASE-17739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926136#comment-15926136 ] Anoop Sam John commented on HBASE-17739: Yes. Once we complete the 64 KB bucket slots, we will convert the 512 KB slots into 64 KB (Bucket will have more slots then and each slot size is reduced) and use. The #2 case is said considering this way only. The point is say when we have blocks of size 2.1 KB and the slot size is 4, each of the block addition waste almost 50% memory. The HP use case which Stack originally described also with low block size. The impact of #2 is more when actual block size itself is more. U can see with every block size bucket, we give some extra size to accommodate block header or so and also block size boundary is not a strict one.. The brainstorming point here is any way we can avoid/reduce this. Hope u got the intent correctly now. > BucketCache is inefficient/wasteful/dumb in its bucket allocations > -- > > Key: HBASE-17739 > URL: https://issues.apache.org/jira/browse/HBASE-17739 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > By default we allocate 14 buckets with sizes from 5K to 513K. If lots of heap > given over to bucketcache and say no allocattions made for a particular > bucket size, this means we have a bunch of the bucketcache that just goes > idle/unused. > For example, say heap is 100G. We'll divide it up among the sizes. If say we > only ever do 5k records, then most of the cache will go unused while the > allocation for 5k objects will see churn. > Here is an old note of [~anoop.hbase]'s' from a conversation on bucket cache > we had offlist that describes the issue: > "By default we have those 14 buckets with size range of 5K to 513K. > All sizes will have one bucket (with size 513*4) each except the > last size.. ie. 513K sized many buckets will be there. If we keep on > writing only same sized blocks, we may loose all in btw sized buckets. > Say we write only 4K sized blocks. We will 1st fill the bucket in 5K > size. There is only one such bucket. Once this is filled, we will try > to grab a complete free bucket from other sizes.. But we can not take > it from 9K... 385K sized ones as there is only ONE bucket for these > sizes. We will take only from 513 size.. There are many in that... > So we will eventually take all the buckets from 513 except the last > one.. Ya it has to keep at least one in evey size.. So we will > loose these much size.. They are of no use." > We should set the size type on the fly as the records come in. > Or better, we should choose record size on the fly. Here is another comment > from [~anoop.hbase]: > "The second is the biggest contributor. Suppose instead of 4K > sized blocks, the user has 2 K sized blocks.. When we write a block to > bucket slot, we will reserve size equal to the allocated size for that block. > So when we write 2K sized blocks (may be actual size a bit more than > 2K ) we will take 5K with each of the block. So u can see that we are > loosing ~3K with every block. Means we are loosing more than half." > He goes on: "If am 100% sure that all my table having 2K HFile block size, I > need to give this config a value 3 * 1024 (Exact 2 K if I give there may be > again problem! That is another story we need to see how we can give > more guarantee for the block size restriction HBASE-15248).. So here also > ~1K loose for every 2K.. So some thing like a 30% loose !!! :-(“" > So, we should figure the record sizes ourselves on the fly. > Anything less has us wasting loads of cache space, nvm inefficiences lost > because of how we serialize base types to cache. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926126#comment-15926126 ] Anoop Sam John commented on HBASE-16438: Not sure then CAS way can work. Might need a lock unlock way only. Because ya one thread succed in getting CAS success/lock for calling ChunkCreator. It will only make the call.. There is a time gap for the actual chunk create work. During this time also, the other threads have to wait. So ya tryLock () unlock way may be needed.. Need to carefully test for the perf implication of any such change (Using a JMH) > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-16438_1.patch, HBASE-16438.patch, > MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17740) Correct the semantic of batch and partial for async client
[
https://issues.apache.org/jira/browse/HBASE-17740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926123#comment-15926123
]
Hadoop QA commented on HBASE-17740:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 28s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 16s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 100m 36s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 149m 55s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12858857/HBASE-17740-v1.patch |
| JIRA Issue | HBASE-17740 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d075aa3123fd 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / a49bc58 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6105/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6105/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Correct the semantic of batch and partial for
[jira] [Commented] (HBASE-16290) Dump summary of callQueue content; can help debugging
[ https://issues.apache.org/jira/browse/HBASE-16290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926108#comment-15926108 ] Sreeram Venkatasubramanian commented on HBASE-16290: [~saint@gmail.com], Is it OK to include the stats as part of 'Debug Dump' of the Master/Regionserver UI ? > Dump summary of callQueue content; can help debugging > - > > Key: HBASE-16290 > URL: https://issues.apache.org/jira/browse/HBASE-16290 > Project: HBase > Issue Type: Bug > Components: Operability >Reporter: stack >Assignee: Sreeram Venkatasubramanian >Priority: Critical > Labels: beginner > > Being able to get a clue what is in a backedup callQueue could give insight > on what is going on on a jacked server. Just needs to summarize count, sizes, > call types. Useful debugging. In a servlet? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17739) BucketCache is inefficient/wasteful/dumb in its bucket allocations
[ https://issues.apache.org/jira/browse/HBASE-17739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925985#comment-15925985 ] ramkrishna.s.vasudevan commented on HBASE-17739: AFter reading the code, I think in case of #1 - yes the buckets in between are not used efficiently when we constantly get 64K buckets because all other intermediate buckets upto 513K size are only one in number. so we will not use them. But once that 513k buckets are getting used up I think we won't waste it instead we will convert those 513k to 64k sized buckets right? So for #2 once we start getting on avg some XKB buckets I think after that overall we wil be allowing the bucket to grow in an uniform way and the wastage should be getting normalized. Infact while freeing the block also - say all the 513 blocks are getting converted to 64k blocks we will keep freeing those blocks and add it to the 64k block count only? I may be wrong. Pls do correct me if I am missing some thing here. I think this problem is more prominent when overall our block sizes keep varying and I think that is where for encoded blocks they try to come up with a standardisation so that after encoding also the ratio is maintained across blocks. > BucketCache is inefficient/wasteful/dumb in its bucket allocations > -- > > Key: HBASE-17739 > URL: https://issues.apache.org/jira/browse/HBASE-17739 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > By default we allocate 14 buckets with sizes from 5K to 513K. If lots of heap > given over to bucketcache and say no allocattions made for a particular > bucket size, this means we have a bunch of the bucketcache that just goes > idle/unused. > For example, say heap is 100G. We'll divide it up among the sizes. If say we > only ever do 5k records, then most of the cache will go unused while the > allocation for 5k objects will see churn. > Here is an old note of [~anoop.hbase]'s' from a conversation on bucket cache > we had offlist that describes the issue: > "By default we have those 14 buckets with size range of 5K to 513K. > All sizes will have one bucket (with size 513*4) each except the > last size.. ie. 513K sized many buckets will be there. If we keep on > writing only same sized blocks, we may loose all in btw sized buckets. > Say we write only 4K sized blocks. We will 1st fill the bucket in 5K > size. There is only one such bucket. Once this is filled, we will try > to grab a complete free bucket from other sizes.. But we can not take > it from 9K... 385K sized ones as there is only ONE bucket for these > sizes. We will take only from 513 size.. There are many in that... > So we will eventually take all the buckets from 513 except the last > one.. Ya it has to keep at least one in evey size.. So we will > loose these much size.. They are of no use." > We should set the size type on the fly as the records come in. > Or better, we should choose record size on the fly. Here is another comment > from [~anoop.hbase]: > "The second is the biggest contributor. Suppose instead of 4K > sized blocks, the user has 2 K sized blocks.. When we write a block to > bucket slot, we will reserve size equal to the allocated size for that block. > So when we write 2K sized blocks (may be actual size a bit more than > 2K ) we will take 5K with each of the block. So u can see that we are > loosing ~3K with every block. Means we are loosing more than half." > He goes on: "If am 100% sure that all my table having 2K HFile block size, I > need to give this config a value 3 * 1024 (Exact 2 K if I give there may be > again problem! That is another story we need to see how we can give > more guarantee for the block size restriction HBASE-15248).. So here also > ~1K loose for every 2K.. So some thing like a 30% loose !!! :-(“" > So, we should figure the record sizes ourselves on the fly. > Anything less has us wasting loads of cache space, nvm inefficiences lost > because of how we serialize base types to cache. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17790) Mark ReplicationAdmin's peerAdded and listReplicationPeers as Deprecated
[ https://issues.apache.org/jira/browse/HBASE-17790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925975#comment-15925975 ] Ashish Singhi commented on HBASE-17790: --- Can you add java doc for the user pointing which method to use instead of this deprecated method. Otherwise +1 if Hadoop QA is green. > Mark ReplicationAdmin's peerAdded and listReplicationPeers as Deprecated > > > Key: HBASE-17790 > URL: https://issues.apache.org/jira/browse/HBASE-17790 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17790.master.001.patch > > > Now most of public method in ReplicationAdmin have been moved to Admin and > marked as Deprecated. And peerAdded and listReplicationPeers method need be > marked as Deprecated, too. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17790) Mark ReplicationAdmin's peerAdded and listReplicationPeers as Deprecated
[ https://issues.apache.org/jira/browse/HBASE-17790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17790: --- Attachment: HBASE-17790.master.001.patch > Mark ReplicationAdmin's peerAdded and listReplicationPeers as Deprecated > > > Key: HBASE-17790 > URL: https://issues.apache.org/jira/browse/HBASE-17790 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17790.master.001.patch > > > Now most of public method in ReplicationAdmin have been moved to Admin and > marked as Deprecated. And peerAdded and listReplicationPeers method need be > marked as Deprecated, too. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17790) Mark ReplicationAdmin's peerAdded and listReplicationPeers as Deprecated
Guanghao Zhang created HBASE-17790: -- Summary: Mark ReplicationAdmin's peerAdded and listReplicationPeers as Deprecated Key: HBASE-17790 URL: https://issues.apache.org/jira/browse/HBASE-17790 Project: HBase Issue Type: Sub-task Reporter: Guanghao Zhang Assignee: Guanghao Zhang Priority: Minor Now most of public method in ReplicationAdmin have been moved to Admin and marked as Deprecated. And peerAdded and listReplicationPeers method need be marked as Deprecated, too. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925877#comment-15925877 ] Anastasia Braginsky commented on HBASE-16438: - I meant to let only one thread from any MSLAB to ask ChunkCreator for the chunk allocation. I was talking about CAS (or any other sunchronization) inside MSLAB. So this fight remains within MSLAB. > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-16438_1.patch, HBASE-16438.patch, > MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17669) Implement async mergeRegion/splitRegion methods.
[ https://issues.apache.org/jira/browse/HBASE-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925871#comment-15925871 ] Duo Zhang commented on HBASE-17669: --- The logic is a bit complicated, I need more time. Sorry for the delay. > Implement async mergeRegion/splitRegion methods. > > > Key: HBASE-17669 > URL: https://issues.apache.org/jira/browse/HBASE-17669 > Project: HBase > Issue Type: Sub-task > Components: Admin, asyncclient, Client >Affects Versions: 2.0.0 >Reporter: Zheng Hu >Assignee: Zheng Hu > Fix For: 2.0.0 > > Attachments: HBASE-17669.v1.patch, HBASE-17669.v2.patch, > HBASE-17669.v3.patch, HBASE-17669.v3.patch > > > RT -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925863#comment-15925863 ] ramkrishna.s.vasudevan commented on HBASE-16438: The cyclic reference is with the current code where if we use chunkCreator in MSLAB and wrap the pool inside chunkCreator. But i think that can be managed if we allow the pool itself to reside inside the ChunkCreator and the HRS also will only instantiate the ChunkCreator and internally the chunkCreator creates and manages the pool. For the 2nd one, if we make the CAS inside chunkCreator then all the MSLAB should wait for getting a successful chunk. Where is now it is within one MSLAB there is a fight. So with in a MSLAB we get a chunk and two threads compete to make it as the curChunk. If we move it to chunkCreator two different threads on two different MSLAB itself need to wait for one to get the chunk and then create for other one. Am saying for the non-pooled cases. > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-16438_1.patch, HBASE-16438.patch, > MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17740) Correct the semantic of batch and partial for async client
[ https://issues.apache.org/jira/browse/HBASE-17740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925862#comment-15925862 ] Duo Zhang commented on HBASE-17740: --- Yeah I forgot to add category for the new UT... Upload v1 to address the problem. Let's wait for the pre commit result. > Correct the semantic of batch and partial for async client > -- > > Key: HBASE-17740 > URL: https://issues.apache.org/jira/browse/HBASE-17740 > Project: HBase > Issue Type: Sub-task > Components: asyncclient, Client, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17740.patch, HBASE-17740-v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17740) Correct the semantic of batch and partial for async client
[ https://issues.apache.org/jira/browse/HBASE-17740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17740: -- Attachment: HBASE-17740-v1.patch > Correct the semantic of batch and partial for async client > -- > > Key: HBASE-17740 > URL: https://issues.apache.org/jira/browse/HBASE-17740 > Project: HBase > Issue Type: Sub-task > Components: asyncclient, Client, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17740.patch, HBASE-17740-v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925855#comment-15925855 ] ramkrishna.s.vasudevan commented on HBASE-15314: [~zjushch] I think there are no objections and I think you can go ahead and commit the patch if you have the time to do it. > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, > HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, > HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, > HBASE-15314-v6.patch, HBASE-15314-v7.patch, HBASE-15314-v8.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17740) Correct the semantic of batch and partial for async client
[ https://issues.apache.org/jira/browse/HBASE-17740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925849#comment-15925849 ] Phil Yang commented on HBASE-17740: --- +1 Is failed test related? > Correct the semantic of batch and partial for async client > -- > > Key: HBASE-17740 > URL: https://issues.apache.org/jira/browse/HBASE-17740 > Project: HBase > Issue Type: Sub-task > Components: asyncclient, Client, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17740.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17765) Reviving the merge possibility in the CompactingMemStore
[ https://issues.apache.org/jira/browse/HBASE-17765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925820#comment-15925820 ] Anastasia Braginsky commented on HBASE-17765: - I am going to upload patch later with other changes. closeSuffix and updateRegionSize booleans are not logically related. closeSuffix is saying whether to close the MSLABs of the removed segments (shouldn't be done for merge) updateRegionSize is saying whether to inform the region about the size change (shouldn't be done on snapshot, before snapshot fully flushed to disk) > Reviving the merge possibility in the CompactingMemStore > > > Key: HBASE-17765 > URL: https://issues.apache.org/jira/browse/HBASE-17765 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-17765-V01.patch > > > According to the new performance results presented in the HBASE-16417 we see > that the read latency of the 90th percentile of the BASIC policy is too big > due to the need to traverse through too many segments in the pipeline. In > this JIRA we correct the bug in the merge sizing calculations and allow > pipeline size threshold to be a configurable parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
