[jira] [Comment Edited] (HBASE-16584) Backport the new ipc implementation in HBASE-16432 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898887#comment-15898887 ] Duo Zhang edited comment on HBASE-16584 at 3/7/17 7:49 AM: --- A huge patch... Next I will test the performance compare to the old implementation. was (Author: apache9): A hugh patch... > Backport the new ipc implementation in HBASE-16432 to branch-1 > -- > > Key: HBASE-16584 > URL: https://issues.apache.org/jira/browse/HBASE-16584 > Project: HBase > Issue Type: Task > Components: Client, IPC/RPC >Affects Versions: 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 1.4.0 > > Attachments: HBASE-16584-branch-1.patch > > > Two problems. > First, as RpcCliemtImpl is the default implementation on branch-1, we need to > confirm that the modification on master does not make it slower. I'm not sure > but I used a big lock to protect everything in the new implementation, so it > may have bad impact on performance. > Second, some configurations of the old AsyncRpcClient is gone. Such as > "hbase.rpc.client.threads.max" and "hbase.rpc.client.nativetransport". Now > You could pass a EventLoopGroup object directly through a helper class which > makes it more flexible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16584) Backport the new ipc implementation in HBASE-16432 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16584: -- Status: Patch Available (was: Open) > Backport the new ipc implementation in HBASE-16432 to branch-1 > -- > > Key: HBASE-16584 > URL: https://issues.apache.org/jira/browse/HBASE-16584 > Project: HBase > Issue Type: Task > Components: Client, IPC/RPC >Affects Versions: 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 1.4.0 > > Attachments: HBASE-16584-branch-1.patch > > > Two problems. > First, as RpcCliemtImpl is the default implementation on branch-1, we need to > confirm that the modification on master does not make it slower. I'm not sure > but I used a big lock to protect everything in the new implementation, so it > may have bad impact on performance. > Second, some configurations of the old AsyncRpcClient is gone. Such as > "hbase.rpc.client.threads.max" and "hbase.rpc.client.nativetransport". Now > You could pass a EventLoopGroup object directly through a helper class which > makes it more flexible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16584) Backport the new ipc implementation in HBASE-16432 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16584: -- Attachment: HBASE-16584-branch-1.patch A hugh patch... > Backport the new ipc implementation in HBASE-16432 to branch-1 > -- > > Key: HBASE-16584 > URL: https://issues.apache.org/jira/browse/HBASE-16584 > Project: HBase > Issue Type: Task > Components: Client, IPC/RPC >Affects Versions: 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 1.4.0 > > Attachments: HBASE-16584-branch-1.patch > > > Two problems. > First, as RpcCliemtImpl is the default implementation on branch-1, we need to > confirm that the modification on master does not make it slower. I'm not sure > but I used a big lock to protect everything in the new implementation, so it > may have bad impact on performance. > Second, some configurations of the old AsyncRpcClient is gone. Such as > "hbase.rpc.client.threads.max" and "hbase.rpc.client.nativetransport". Now > You could pass a EventLoopGroup object directly through a helper class which > makes it more flexible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16584) Backport the new ipc implementation in HBASE-16432 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16584: -- Affects Version/s: 1.4.0 Fix Version/s: 1.4.0 Component/s: IPC/RPC Client > Backport the new ipc implementation in HBASE-16432 to branch-1 > -- > > Key: HBASE-16584 > URL: https://issues.apache.org/jira/browse/HBASE-16584 > Project: HBase > Issue Type: Task > Components: Client, IPC/RPC >Affects Versions: 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 1.4.0 > > > Two problems. > First, as RpcCliemtImpl is the default implementation on branch-1, we need to > confirm that the modification on master does not make it slower. I'm not sure > but I used a big lock to protect everything in the new implementation, so it > may have bad impact on performance. > Second, some configurations of the old AsyncRpcClient is gone. Such as > "hbase.rpc.client.threads.max" and "hbase.rpc.client.nativetransport". Now > You could pass a EventLoopGroup object directly through a helper class which > makes it more flexible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-16584) Backport the new ipc implementation in HBASE-16432 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-16584: - Assignee: Duo Zhang > Backport the new ipc implementation in HBASE-16432 to branch-1 > -- > > Key: HBASE-16584 > URL: https://issues.apache.org/jira/browse/HBASE-16584 > Project: HBase > Issue Type: Task > Components: Client, IPC/RPC >Affects Versions: 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 1.4.0 > > > Two problems. > First, as RpcCliemtImpl is the default implementation on branch-1, we need to > confirm that the modification on master does not make it slower. I'm not sure > but I used a big lock to protect everything in the new implementation, so it > may have bad impact on performance. > Second, some configurations of the old AsyncRpcClient is gone. Such as > "hbase.rpc.client.threads.max" and "hbase.rpc.client.nativetransport". Now > You could pass a EventLoopGroup object directly through a helper class which > makes it more flexible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17600) Implement get/create/modify/delete/list namespace admin operations
[ https://issues.apache.org/jira/browse/HBASE-17600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17600: --- Attachment: HBASE-17600.master.004.patch > Implement get/create/modify/delete/list namespace admin operations > -- > > Key: HBASE-17600 > URL: https://issues.apache.org/jira/browse/HBASE-17600 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17600.master.001.patch, > HBASE-17600.master.002.patch, HBASE-17600.master.003.patch, > HBASE-17600.master.003.patch, HBASE-17600.master.004.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-12790) Support fairness across parallelized scans
[ https://issues.apache.org/jira/browse/HBASE-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898857#comment-15898857 ] Andrew Purtell commented on HBASE-12790: [~samarthjain] Pluggable RPC scheduler plus API to set labels (or as you said 'tags') on the RPC messages of client ops, presumably the ops themselves, adding to Operation perhaps. > Support fairness across parallelized scans > -- > > Key: HBASE-12790 > URL: https://issues.apache.org/jira/browse/HBASE-12790 > Project: HBase > Issue Type: New Feature >Reporter: James Taylor >Assignee: ramkrishna.s.vasudevan > Labels: Phoenix > Attachments: AbstractRoundRobinQueue.java, HBASE-12790_1.patch, > HBASE-12790_5.patch, HBASE-12790_callwrapper.patch, HBASE-12790.patch, > HBASE-12790_trunk_1.patch, PHOENIX_4.5.3-HBase-0.98-2317-SNAPSHOT.zip > > > Some HBase clients parallelize the execution of a scan to reduce latency in > getting back results. This can lead to starvation with a loaded cluster and > interleaved scans, since the RPC queue will be ordered and processed on a > FIFO basis. For example, if there are two clients, A & B that submit largish > scans at the same time. Say each scan is broken down into 100 scans by the > client (broken down into equal depth chunks along the row key), and the 100 > scans of client A are queued first, followed immediately by the 100 scans of > client B. In this case, client B will be starved out of getting any results > back until the scans for client A complete. > One solution to this is to use the attached AbstractRoundRobinQueue instead > of the standard FIFO queue. The queue to be used could be (maybe it already > is) configurable based on a new config parameter. Using this queue would > require the client to have the same identifier for all of the 100 parallel > scans that represent a single logical scan from the clients point of view. > With this information, the round robin queue would pick off a task from the > queue in a round robin fashion (instead of a strictly FIFO manner) to prevent > starvation over interleaved parallelized scans. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-12790) Support fairness across parallelized scans
[
https://issues.apache.org/jira/browse/HBASE-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898836#comment-15898836
]
Samarth Jain commented on HBASE-12790:
--
[~lhofhansl] - just having a pluggable RPCScheduler won't do it. Assuming we
are targeting only queries for now, we need to be able to tag/mark scans
belonging to a query with the same identifier. This information will be then
used by the round robin queue to do the round-robin of groups thing.
{code}
@Override
public boolean dispatch(final CallRunner callTask) throws
InterruptedException {
roundRobinQueue.offer(callTask);
}
{code}
> Support fairness across parallelized scans
> --
>
> Key: HBASE-12790
> URL: https://issues.apache.org/jira/browse/HBASE-12790
> Project: HBase
> Issue Type: New Feature
>Reporter: James Taylor
>Assignee: ramkrishna.s.vasudevan
> Labels: Phoenix
> Attachments: AbstractRoundRobinQueue.java, HBASE-12790_1.patch,
> HBASE-12790_5.patch, HBASE-12790_callwrapper.patch, HBASE-12790.patch,
> HBASE-12790_trunk_1.patch, PHOENIX_4.5.3-HBase-0.98-2317-SNAPSHOT.zip
>
>
> Some HBase clients parallelize the execution of a scan to reduce latency in
> getting back results. This can lead to starvation with a loaded cluster and
> interleaved scans, since the RPC queue will be ordered and processed on a
> FIFO basis. For example, if there are two clients, A & B that submit largish
> scans at the same time. Say each scan is broken down into 100 scans by the
> client (broken down into equal depth chunks along the row key), and the 100
> scans of client A are queued first, followed immediately by the 100 scans of
> client B. In this case, client B will be starved out of getting any results
> back until the scans for client A complete.
> One solution to this is to use the attached AbstractRoundRobinQueue instead
> of the standard FIFO queue. The queue to be used could be (maybe it already
> is) configurable based on a new config parameter. Using this queue would
> require the client to have the same identifier for all of the 100 parallel
> scans that represent a single logical scan from the clients point of view.
> With this information, the round robin queue would pick off a task from the
> queue in a round robin fashion (instead of a strictly FIFO manner) to prevent
> starvation over interleaved parallelized scans.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-15484) Correct the semantic of batch and partial
[ https://issues.apache.org/jira/browse/HBASE-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898809#comment-15898809 ] Phil Yang commented on HBASE-15484: --- bq. Want to amend the javadoc on mayHaveMoreCellsInRow method ... Oh I missed that, we can do it in HBASE-17740 > Correct the semantic of batch and partial > - > > Key: HBASE-15484 > URL: https://issues.apache.org/jira/browse/HBASE-15484 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Affects Versions: 2.0.0, 1.4.0 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15484.branch-1.v01.patch, HBASE-15484.v05.patch, > HBASE-15484.v06.patch, HBASE-15484-v1.patch, HBASE-15484-v2.patch, > HBASE-15484-v3.patch, HBASE-15484-v4.patch > > > Follow-up to HBASE-15325, as discussed, the meaning of setBatch and > setAllowPartialResults should not be same. We should not regard setBatch as > setAllowPartialResults. > Now we deprecated isPartial and use mayHaveMoreCellsInRow. If it returns > false, current Result must be the last one of this row. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17742) Reverse scan with empty start row returns incorrect result
[ https://issues.apache.org/jira/browse/HBASE-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898808#comment-15898808 ] Duo Zhang commented on HBASE-17742: --- I think empty start row for reversed scan means start from the last? Am I wrong? Then how do you scan the whole table reversed if we apply the patch here? > Reverse scan with empty start row returns incorrect result > -- > > Key: HBASE-17742 > URL: https://issues.apache.org/jira/browse/HBASE-17742 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0 >Reporter: Yu Li >Assignee: Yu Li > Attachments: HBASE-17742.patch > > > As titled, reverse scan with empty start row should return empty result but > doesn't in latest master code base. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15484) Correct the semantic of batch and partial
[ https://issues.apache.org/jira/browse/HBASE-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-15484: -- Release Note: Now setBatch doesn't mean setAllowPartialResult(true) If user setBatch(5) and rpc returns 3+5+5+5+3 cells, we should return 5+5+5+5+1 to user. We deprecated isPartial and use mayHaveMoreCellsInRow. If it returns false, current Result must be the last one of this row. was: We do not regard setBatch as setAllowPartialResults any more. We deprecated isPartial and use mayHaveMoreCellsInRow. If it returns false, current Result must be the last one of this row. > Correct the semantic of batch and partial > - > > Key: HBASE-15484 > URL: https://issues.apache.org/jira/browse/HBASE-15484 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Affects Versions: 2.0.0, 1.4.0 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15484.branch-1.v01.patch, HBASE-15484.v05.patch, > HBASE-15484.v06.patch, HBASE-15484-v1.patch, HBASE-15484-v2.patch, > HBASE-15484-v3.patch, HBASE-15484-v4.patch > > > Follow-up to HBASE-15325, as discussed, the meaning of setBatch and > setAllowPartialResults should not be same. We should not regard setBatch as > setAllowPartialResults. > Now we deprecated isPartial and use mayHaveMoreCellsInRow. If it returns > false, current Result must be the last one of this row. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15484) Correct the semantic of batch and partial
[
https://issues.apache.org/jira/browse/HBASE-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898805#comment-15898805
]
Duo Zhang commented on HBASE-15484:
---
{quote}
hasMoreCellsInRow was never in a release? Thats why it ok to change the method
name?
{quote}
Yeah, no release yet, luckily...
> Correct the semantic of batch and partial
> -
>
> Key: HBASE-15484
> URL: https://issues.apache.org/jira/browse/HBASE-15484
> Project: HBase
> Issue Type: Sub-task
> Components: Client, scan
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Phil Yang
>Assignee: Phil Yang
>Priority: Blocker
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-15484.branch-1.v01.patch, HBASE-15484.v05.patch,
> HBASE-15484.v06.patch, HBASE-15484-v1.patch, HBASE-15484-v2.patch,
> HBASE-15484-v3.patch, HBASE-15484-v4.patch
>
>
> Follow-up to HBASE-15325, as discussed, the meaning of setBatch and
> setAllowPartialResults should not be same. We should not regard setBatch as
> setAllowPartialResults.
> Now we deprecated isPartial and use mayHaveMoreCellsInRow. If it returns
> false, current Result must be the last one of this row.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-15484) Correct the semantic of batch and partial
[ https://issues.apache.org/jira/browse/HBASE-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898802#comment-15898802 ] stack commented on HBASE-15484: --- Patch looks great. +1 Would it be good to add this to release note [~yangzhe1991]? * If user setBatch(5) and rpc returns 3+5+5+5+3 cells, we should return 5+5+5+5+1 to user. * setBatch doesn't mean setAllowPartialResult(true) hasMoreCellsInRow was never in a release? Thats why it ok to change the method name? Want to amend the javadoc on mayHaveMoreCellsInRow method w/ a quick amendment so say you can't be always sure you are at the end of a row (add in 'may' to fix this bit in javadoc on the new method... "...True means there are be more cells for the current row") Good stuff. > Correct the semantic of batch and partial > - > > Key: HBASE-15484 > URL: https://issues.apache.org/jira/browse/HBASE-15484 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Affects Versions: 2.0.0, 1.4.0 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15484.branch-1.v01.patch, HBASE-15484.v05.patch, > HBASE-15484.v06.patch, HBASE-15484-v1.patch, HBASE-15484-v2.patch, > HBASE-15484-v3.patch, HBASE-15484-v4.patch > > > Follow-up to HBASE-15325, as discussed, the meaning of setBatch and > setAllowPartialResults should not be same. We should not regard setBatch as > setAllowPartialResults. > Now we deprecated isPartial and use mayHaveMoreCellsInRow. If it returns > false, current Result must be the last one of this row. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17532) Replace explicit type with diamond operator
[
https://issues.apache.org/jira/browse/HBASE-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898796#comment-15898796
]
Hadoop QA commented on HBASE-17532:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 10s {color}
| {color:red} HBASE-17532 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856438/HBASE-17532.master.002.patch
|
| JIRA Issue | HBASE-17532 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5977/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Replace explicit type with diamond operator
> ---
>
> Key: HBASE-17532
> URL: https://issues.apache.org/jira/browse/HBASE-17532
> Project: HBase
> Issue Type: Improvement
> Components: build
>Reporter: Jan Hentschel
>Assignee: Jan Hentschel
>Priority: Trivial
> Fix For: 2.0.0
>
> Attachments: HBASE-17532.master.001.patch,
> HBASE-17532.master.002.patch, HBASE-17532.master.002.patch
>
>
> Because HBase uses Java 8 explicit types can be replaced with the diamond
> operator.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17532) Replace explicit type with diamond operator
[ https://issues.apache.org/jira/browse/HBASE-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17532: -- Attachment: HBASE-17532.master.002.patch Retry to be sure. Failed test passes locally. > Replace explicit type with diamond operator > --- > > Key: HBASE-17532 > URL: https://issues.apache.org/jira/browse/HBASE-17532 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17532.master.001.patch, > HBASE-17532.master.002.patch, HBASE-17532.master.002.patch > > > Because HBase uses Java 8 explicit types can be replaced with the diamond > operator. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17739) BucketCache is inefficient/wasteful/dumb in its bucket allocations
[ https://issues.apache.org/jira/browse/HBASE-17739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898786#comment-15898786 ] stack commented on HBASE-17739: --- bq. How about making the encoded blocks have similar size? Tell us more [~zjushch]? (Thanks for all the help on bucket cache) > BucketCache is inefficient/wasteful/dumb in its bucket allocations > -- > > Key: HBASE-17739 > URL: https://issues.apache.org/jira/browse/HBASE-17739 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > By default we allocate 14 buckets with sizes from 5K to 513K. If lots of heap > given over to bucketcache and say no allocattions made for a particular > bucket size, this means we have a bunch of the bucketcache that just goes > idle/unused. > For example, say heap is 100G. We'll divide it up among the sizes. If say we > only ever do 5k records, then most of the cache will go unused while the > allocation for 5k objects will see churn. > Here is an old note of [~anoop.hbase]'s' from a conversation on bucket cache > we had offlist that describes the issue: > "By default we have those 14 buckets with size range of 5K to 513K. > All sizes will have one bucket (with size 513*4) each except the > last size.. ie. 513K sized many buckets will be there. If we keep on > writing only same sized blocks, we may loose all in btw sized buckets. > Say we write only 4K sized blocks. We will 1st fill the bucket in 5K > size. There is only one such bucket. Once this is filled, we will try > to grab a complete free bucket from other sizes.. But we can not take > it from 9K... 385K sized ones as there is only ONE bucket for these > sizes. We will take only from 513 size.. There are many in that... > So we will eventually take all the buckets from 513 except the last > one.. Ya it has to keep at least one in evey size.. So we will > loose these much size.. They are of no use." > We should set the size type on the fly as the records come in. > Or better, we should choose record size on the fly. Here is another comment > from [~anoop.hbase]: > "The second is the biggest contributor. Suppose instead of 4K > sized blocks, the user has 2 K sized blocks.. When we write a block to > bucket slot, we will reserve size equal to the allocated size for that block. > So when we write 2K sized blocks (may be actual size a bit more than > 2K ) we will take 5K with each of the block. So u can see that we are > loosing ~3K with every block. Means we are loosing more than half." > He goes on: "If am 100% sure that all my table having 2K HFile block size, I > need to give this config a value 3 * 1024 (Exact 2 K if I give there may be > again problem! That is another story we need to see how we can give > more guarantee for the block size restriction HBASE-15248).. So here also > ~1K loose for every 2K.. So some thing like a 30% loose !!! :-(“" > So, we should figure the record sizes ourselves on the fly. > Anything less has us wasting loads of cache space, nvm inefficiences lost > because of how we serialize base types to cache. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17716) Formalize Scan Metric names
[ https://issues.apache.org/jira/browse/HBASE-17716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17716: -- Attachment: HBASE-17716_v2.patch Retry to see if test failure related. Don't think it possilble. Patch looks good. I'll commit and fix the white space if retry shows it good Thanks Karan. > Formalize Scan Metric names > --- > > Key: HBASE-17716 > URL: https://issues.apache.org/jira/browse/HBASE-17716 > Project: HBase > Issue Type: Bug > Components: metrics >Reporter: Karan Mehta >Assignee: Karan Mehta >Priority: Minor > Attachments: HBASE-17716.patch, HBASE-17716_v2.patch, > HBASE-17716_v2.patch > > > HBase provides various metrics through the API's exposed by ScanMetrics > class. > The JIRA PHOENIX-3248 requires them to be surfaced through the Phoenix > Metrics API. Currently these metrics are referred via hard-coded strings, > which are not formal and can break the Phoenix API. Hence we need to refactor > the code to assign enums for these metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache
[
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898772#comment-15898772
]
Hadoop QA commented on HBASE-15314:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 1s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 126m 35s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 165m 59s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856419/HBASE-15314-v6.patch |
| JIRA Issue | HBASE-15314 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 5a6e87738b81 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 81cb298 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5975/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/5975/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5975/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5975/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
>
[jira] [Commented] (HBASE-17532) Replace explicit type with diamond operator
[
https://issues.apache.org/jira/browse/HBASE-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898761#comment-15898761
]
Hadoop QA commented on HBASE-17532:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 427 new or modified
test files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
7s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 10s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 5m
40s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 38s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 29s
{color} | {color:red} hbase-prefix-tree in master has 1 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 6s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 8m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 5m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
39m 9s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 17m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} hbase-hadoop2-compat generated 0 new + 1 unchanged - 1
fixed = 1 total (was 2) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} hbase-prefix-tree in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} hbase-thrift in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
|
[jira] [Commented] (HBASE-17716) Formalize Scan Metric names
[
https://issues.apache.org/jira/browse/HBASE-17716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898757#comment-15898757
]
Hadoop QA commented on HBASE-17716:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 38s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 18s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 109m 49s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 161m 44s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.balancer.TestStochasticLoadBalancer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856417/HBASE-17716_v2.patch |
| JIRA Issue | HBASE-17716 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 07cc81b488d4 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 81cb298 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/5974/artifact/patchprocess/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5974/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test
[jira] [Commented] (HBASE-17742) Reverse scan with empty start row returns incorrect result
[ https://issues.apache.org/jira/browse/HBASE-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898756#comment-15898756 ] Yu Li commented on HBASE-17742: --- [~Apache9] mind take a look here? Thanks. > Reverse scan with empty start row returns incorrect result > -- > > Key: HBASE-17742 > URL: https://issues.apache.org/jira/browse/HBASE-17742 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0 >Reporter: Yu Li >Assignee: Yu Li > Attachments: HBASE-17742.patch > > > As titled, reverse scan with empty start row should return empty result but > doesn't in latest master code base. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17742) Reverse scan with empty start row returns incorrect result
[ https://issues.apache.org/jira/browse/HBASE-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-17742: -- Attachment: HBASE-17742.patch Uploading an initial patch, the newly added UT case could reveal the issue. > Reverse scan with empty start row returns incorrect result > -- > > Key: HBASE-17742 > URL: https://issues.apache.org/jira/browse/HBASE-17742 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0 >Reporter: Yu Li >Assignee: Yu Li > Attachments: HBASE-17742.patch > > > As titled, reverse scan with empty start row should return empty result but > doesn't in latest master code base. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17742) Reverse scan with empty start row returns incorrect result
Yu Li created HBASE-17742: - Summary: Reverse scan with empty start row returns incorrect result Key: HBASE-17742 URL: https://issues.apache.org/jira/browse/HBASE-17742 Project: HBase Issue Type: Bug Affects Versions: 2.0 Reporter: Yu Li Assignee: Yu Li As titled, reverse scan with empty start row should return empty result but doesn't in latest master code base. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15248) BLOCKSIZE 4k should result in 4096 bytes on disk; i.e. fit inside a BucketCache 'block' of 4k
[ https://issues.apache.org/jira/browse/HBASE-15248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898738#comment-15898738 ] Hudson commented on HBASE-15248: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2626 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2626/]) Javadoc on what BLOCKSIZE means (related to HBASE-15248) (stack: rev 4beae9a56e2d5d2992d017ba876d18f78415e067) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java > BLOCKSIZE 4k should result in 4096 bytes on disk; i.e. fit inside a > BucketCache 'block' of 4k > - > > Key: HBASE-15248 > URL: https://issues.apache.org/jira/browse/HBASE-15248 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > Chatting w/ a gentleman named Daniel Pol who is messing w/ bucketcache, he > wants blocks to be the size specified in the configuration and no bigger. His > hardware set ups fetches pages of 4k and so a block that has 4k of payload > but has then a header and the header of the next block (which helps figure > whats next when scanning) ends up being 4203 bytes or something, and this > then then translates into two seeks per block fetch. > This issue is about what it would take to stay inside our configured size > boundary writing out blocks. > If not possible, give back better signal on what to do so you could fit > inside a particular constraint. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15871) Memstore flush doesn't finish because of backwardseek() in memstore scanner.
[ https://issues.apache.org/jira/browse/HBASE-15871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898720#comment-15898720 ] Ted Yu commented on HBASE-15871: I don't see this being fixed in branch-1 > Memstore flush doesn't finish because of backwardseek() in memstore scanner. > > > Key: HBASE-15871 > URL: https://issues.apache.org/jira/browse/HBASE-15871 > Project: HBase > Issue Type: Bug > Components: Scanners >Affects Versions: 1.1.2 >Reporter: Jeongdae Kim >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15871_1.patch, HBASE-15871_1.patch, > HBASE-15871_2.patch, HBASE-15871_3.patch, HBASE-15871_4.patch, > HBASE-15871_6.patch, HBASE-15871.branch-1.1.001.patch, > HBASE-15871.branch-1.1.002.patch, HBASE-15871.branch-1.1.003.patch, > HBASE-15871-branch-1.patch, HBASE-15871.patch, memstore_backwardSeek().PNG > > > Sometimes in our production hbase cluster, it takes a long time to finish > memstore flush.( for about more than 30 minutes) > the reason is that a memstore flusher thread calls > StoreScanner.updateReaders(), waits for acquiring a lock that store scanner > holds in StoreScanner.next() and backwardseek() in memstore scanner runs for > a long time. > I think that this condition could occur in reverse scan by the following > process. > 1) create a reversed store scanner by requesting a reverse scan. > 2) flush a memstore in the same HStore. > 3) puts a lot of cells in memstore and memstore is almost full. > 4) call the reverse scanner.next() and re-create all scanners in this store > because all scanners was already closed by 2)'s flush() and backwardseek() > with store's lastTop for all new scanners. > 5) in this status, memstore is almost full by 2) and all cells in memstore > have sequenceID greater than this scanner's readPoint because of 2)'s > flush(). this condition causes searching all cells in memstore, and > seekToPreviousRow() repeatly seach cells that are already searched if a row > has one column. (described this in more detail in a attached file.) > 6) flush a memstore again in the same HStore, and wait until 4-5) process > finished, to update store files in the same HStore after flusing. > I searched HBase jira. and found a similar issue. (HBASE-14497) but, > HBASE-14497's fix can't solve this issue because that fix just changed > recursive call to loop.(and already applied to our HBase version) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15871) Memstore flush doesn't finish because of backwardseek() in memstore scanner.
[ https://issues.apache.org/jira/browse/HBASE-15871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898710#comment-15898710 ] sunghan.suh commented on HBASE-15871: - I'm using HDP 2.3.4 with HBase 1.1.2. Could you apply this patch to this version or give me a guide? > Memstore flush doesn't finish because of backwardseek() in memstore scanner. > > > Key: HBASE-15871 > URL: https://issues.apache.org/jira/browse/HBASE-15871 > Project: HBase > Issue Type: Bug > Components: Scanners >Affects Versions: 1.1.2 >Reporter: Jeongdae Kim >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15871_1.patch, HBASE-15871_1.patch, > HBASE-15871_2.patch, HBASE-15871_3.patch, HBASE-15871_4.patch, > HBASE-15871_6.patch, HBASE-15871.branch-1.1.001.patch, > HBASE-15871.branch-1.1.002.patch, HBASE-15871.branch-1.1.003.patch, > HBASE-15871-branch-1.patch, HBASE-15871.patch, memstore_backwardSeek().PNG > > > Sometimes in our production hbase cluster, it takes a long time to finish > memstore flush.( for about more than 30 minutes) > the reason is that a memstore flusher thread calls > StoreScanner.updateReaders(), waits for acquiring a lock that store scanner > holds in StoreScanner.next() and backwardseek() in memstore scanner runs for > a long time. > I think that this condition could occur in reverse scan by the following > process. > 1) create a reversed store scanner by requesting a reverse scan. > 2) flush a memstore in the same HStore. > 3) puts a lot of cells in memstore and memstore is almost full. > 4) call the reverse scanner.next() and re-create all scanners in this store > because all scanners was already closed by 2)'s flush() and backwardseek() > with store's lastTop for all new scanners. > 5) in this status, memstore is almost full by 2) and all cells in memstore > have sequenceID greater than this scanner's readPoint because of 2)'s > flush(). this condition causes searching all cells in memstore, and > seekToPreviousRow() repeatly seach cells that are already searched if a row > has one column. (described this in more detail in a attached file.) > 6) flush a memstore again in the same HStore, and wait until 4-5) process > finished, to update store files in the same HStore after flusing. > I searched HBase jira. and found a similar issue. (HBASE-14497) but, > HBASE-14497's fix can't solve this issue because that fix just changed > recursive call to loop.(and already applied to our HBase version) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15484) Correct the semantic of batch and partial
[ https://issues.apache.org/jira/browse/HBASE-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-15484: -- Resolution: Fixed Hadoop Flags: Reviewed Release Note: We do not regard setBatch as setAllowPartialResults any more. We deprecated isPartial and use mayHaveMoreCellsInRow. If it returns false, current Result must be the last one of this row. Status: Resolved (was: Patch Available) Pushed to master and branch-1. Thanks all for review. > Correct the semantic of batch and partial > - > > Key: HBASE-15484 > URL: https://issues.apache.org/jira/browse/HBASE-15484 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Affects Versions: 2.0.0, 1.4.0 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15484.branch-1.v01.patch, HBASE-15484.v05.patch, > HBASE-15484.v06.patch, HBASE-15484-v1.patch, HBASE-15484-v2.patch, > HBASE-15484-v3.patch, HBASE-15484-v4.patch > > > Follow-up to HBASE-15325, as discussed, the meaning of setBatch and > setAllowPartialResults should not be same. We should not regard setBatch as > setAllowPartialResults. > Now we deprecated isPartial and use mayHaveMoreCellsInRow. If it returns > false, current Result must be the last one of this row. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16421) Introducing the CellChunkMap as a new additional index variant in the MemStore
[ https://issues.apache.org/jira/browse/HBASE-16421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898684#comment-15898684 ] ramkrishna.s.vasudevan commented on HBASE-16421: [~ebortnik] Since we had done extensive testing during our POC we had infact done all the changes using the experimental ChunkMap also. When I prepare a patch now I will just create the MemstorechunkCreator (which was not there ) and MemstoreChunkCell (new cell type) Later it can be merged with ChunkCellMap impl. If you feel the patch needs more chnges pls feel free to make it and we can take it for commit. > Introducing the CellChunkMap as a new additional index variant in the MemStore > -- > > Key: HBASE-16421 > URL: https://issues.apache.org/jira/browse/HBASE-16421 > Project: HBase > Issue Type: Umbrella >Reporter: Anastasia Braginsky > Attachments: CellChunkMapRevived.pdf, ChunkCell_creation.png, > IntroductiontoNewFlatandCompactMemStore.pdf > > > Follow up for HBASE-14921. This is going to be the umbrella JIRA to include > all the parts of integration of the CellChunkMap to the MemStore. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17728) Create separate build target for Configuration classes
[ https://issues.apache.org/jira/browse/HBASE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17728: --- Attachment: 17728.v5.txt Patch v5 is rebased on HBASE-17741 > Create separate build target for Configuration classes > -- > > Key: HBASE-17728 > URL: https://issues.apache.org/jira/browse/HBASE-17728 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 17728.v1.txt, 17728.v2.txt, 17728.v3.txt, 17728.v4.txt, > 17728.v5.txt > > > User is in security module. > When User::isSecurityEnabled() is added, we need to query Configuration for > security setting. > However, this introduces a circular build dependency: > BUILD FAILED: Cycle found: //connection:connection -> //security:security -> > //core:core -> //connection:connection > This issue is to create separate build target for Configuration which is > depended upon by both core and security modules. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-17741) [C++] rename some files to use (dash) instead of _
[
https://issues.apache.org/jira/browse/HBASE-17741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar resolved HBASE-17741.
---
Resolution: Fixed
Pushed to branch. [~sudeeps], [~xiaobingo] FYI.
> [C++] rename some files to use (dash) instead of _
> ---
>
> Key: HBASE-17741
> URL: https://issues.apache.org/jira/browse/HBASE-17741
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: HBASE-14850
>
> Attachments: hbase-17741_v1.patch
>
>
> Some files are named {{foo_bar.h}}, while almost all of the remaining are
> named like {{foo-bar.h}}. We should stick to using the latter.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17741) [C++] rename some files to use (dash) instead of _
[
https://issues.apache.org/jira/browse/HBASE-17741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-17741:
--
Attachment: hbase-17741_v1.patch
trivial patch.
> [C++] rename some files to use (dash) instead of _
> ---
>
> Key: HBASE-17741
> URL: https://issues.apache.org/jira/browse/HBASE-17741
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: HBASE-14850
>
> Attachments: hbase-17741_v1.patch
>
>
> Some files are named {{foo_bar.h}}, while almost all of the remaining are
> named like {{foo-bar.h}}. We should stick to using the latter.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HBASE-17741) [C++] rename some files to use (dash) instead of _
Enis Soztutar created HBASE-17741:
-
Summary: [C++] rename some files to use (dash) instead of _
Key: HBASE-17741
URL: https://issues.apache.org/jira/browse/HBASE-17741
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: HBASE-14850
Some files are named {{foo_bar.h}}, while almost all of the remaining are named
like {{foo-bar.h}}. We should stick to using the latter.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17717) Incorrect ZK ACL set for HBase superuser
[
https://issues.apache.org/jira/browse/HBASE-17717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898657#comment-15898657
]
Hudson commented on HBASE-17717:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #112 (See
[https://builds.apache.org/job/HBase-1.2-JDK7/112/])
HBASE-17717 Explicitly use "sasl" ACL scheme for hbase superuser (elserj: rev
628490b29e20cb33257d0997b28d14cfd3e9a456)
* (edit)
hbase-client/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKUtil.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
> Incorrect ZK ACL set for HBase superuser
>
>
> Key: HBASE-17717
> URL: https://issues.apache.org/jira/browse/HBASE-17717
> Project: HBase
> Issue Type: Bug
> Components: security, Zookeeper
>Reporter: Shreya Bhat
>Assignee: Josh Elser
> Fix For: 2.0.0, 1.4.0, 1.3.1, 1.1.10, 1.2.6
>
> Attachments: HBASE-17717.001.0.98.patch,
> HBASE-17717.001.branch-1.1.patch, HBASE-17717.001.patch
>
>
> Shreya was doing some testing of a deploy of HBase, verifying that the ZK
> ACLs were actually set as we expect (yay, security).
> She noticed that, in some cases, we were seeing multiple ACLs for the same
> user.
> {noformat}
> 'world,'anyone
> : r
> 'sasl,'hbase
> : cdrwa
> 'sasl,'hbase
> : cdrwa
> {noformat}
> After digging into this (and some insight from the mighty [~enis]), we
> realized that this was happening because of an overridden value for
> {{hbase.superuser}}. However, the ACL value doesn't match what we'd expect to
> see (as hbase.superuser was set to {{cstm-hbase}}).
> After digging into this code, it seems like the {{auth}} ACL scheme in
> ZooKeeper does not work as we expect.
> {code}
> if (superUser != null) {
> acls.add(new ACL(Perms.ALL, new Id("auth", superUser)));
> }
> {code}
> In the above, the {{"auth"}} scheme ignores any provided "subject" in the
> {{Id}} object. It *only* considers the authentication of the current
> connection. As such, our usage of this never actually sets the ACL for the
> superuser correctly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16584) Backport the new ipc implementation in HBASE-16432 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898629#comment-15898629 ] Duo Zhang commented on HBASE-16584: --- Let me pick this up. We decide to upgrade to HBase-1.x late this year and we want to use async client as our internal code base is jdk8 only. So if master and branch-1 can share the same rpc implementation, it is much easier for us to backport the async client stuffs to branch-1 in our internal version. > Backport the new ipc implementation in HBASE-16432 to branch-1 > -- > > Key: HBASE-16584 > URL: https://issues.apache.org/jira/browse/HBASE-16584 > Project: HBase > Issue Type: Task >Reporter: Duo Zhang > > Two problems. > First, as RpcCliemtImpl is the default implementation on branch-1, we need to > confirm that the modification on master does not make it slower. I'm not sure > but I used a big lock to protect everything in the new implementation, so it > may have bad impact on performance. > Second, some configurations of the old AsyncRpcClient is gone. Such as > "hbase.rpc.client.threads.max" and "hbase.rpc.client.nativetransport". Now > You could pass a EventLoopGroup object directly through a helper class which > makes it more flexible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-15314: - Attachment: HBASE-15314-v6.patch Uploaded the latest patch (15314-v6.patch) on review board > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, > HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, > HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, > HBASE-15314-v6.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17698) ReplicationEndpoint choosing sinks
[
https://issues.apache.org/jira/browse/HBASE-17698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898616#comment-15898616
]
Hadoop QA commented on HBASE-17698:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 20s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 92m 35s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 131m 21s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856380/HBASE-17698.patch |
| JIRA Issue | HBASE-17698 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux fc32a5b225e8 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 4beae9a |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5972/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5972/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> ReplicationEndpoint choosing sinks
> --
>
> Key: HBASE-17698
> URL: https://issues.apache.org/jira/browse/HBASE-17698
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0
>Reporter: churro morales
>
[jira] [Commented] (HBASE-17716) Formalize Scan Metric names
[ https://issues.apache.org/jira/browse/HBASE-17716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898614#comment-15898614 ] Karan Mehta commented on HBASE-17716: - Added a patch which appends the metric names strings with the suffix "_METRIC_NAME" and made them as public static final. > Formalize Scan Metric names > --- > > Key: HBASE-17716 > URL: https://issues.apache.org/jira/browse/HBASE-17716 > Project: HBase > Issue Type: Bug > Components: metrics >Reporter: Karan Mehta >Assignee: Karan Mehta >Priority: Minor > Attachments: HBASE-17716.patch, HBASE-17716_v2.patch > > > HBase provides various metrics through the API's exposed by ScanMetrics > class. > The JIRA PHOENIX-3248 requires them to be surfaced through the Phoenix > Metrics API. Currently these metrics are referred via hard-coded strings, > which are not formal and can break the Phoenix API. Hence we need to refactor > the code to assign enums for these metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17716) Formalize Scan Metric names
[ https://issues.apache.org/jira/browse/HBASE-17716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karan Mehta updated HBASE-17716: Attachment: HBASE-17716_v2.patch > Formalize Scan Metric names > --- > > Key: HBASE-17716 > URL: https://issues.apache.org/jira/browse/HBASE-17716 > Project: HBase > Issue Type: Bug > Components: metrics >Reporter: Karan Mehta >Assignee: Karan Mehta >Priority: Minor > Attachments: HBASE-17716.patch, HBASE-17716_v2.patch > > > HBase provides various metrics through the API's exposed by ScanMetrics > class. > The JIRA PHOENIX-3248 requires them to be surfaced through the Phoenix > Metrics API. Currently these metrics are referred via hard-coded strings, > which are not formal and can break the Phoenix API. Hence we need to refactor > the code to assign enums for these metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17734) Guard against possibly copying the qualifier in the ScanDeleteTracker
[
https://issues.apache.org/jira/browse/HBASE-17734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898606#comment-15898606
]
CHIA-PING TSAI commented on HBASE-17734:
Thanks for the help. [~yuzhih...@gmail.com]
> Guard against possibly copying the qualifier in the ScanDeleteTracker
> -
>
> Key: HBASE-17734
> URL: https://issues.apache.org/jira/browse/HBASE-17734
> Project: HBase
> Issue Type: Improvement
>Reporter: CHIA-PING TSAI
>Assignee: CHIA-PING TSAI
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17734.v0.patch, HBASE-17734.v1.patch,
> HBASE-17734.v2.patch
>
>
> If the input cell is ByteBufferKeyValue, the
> ByteBufferKeyValue#getQualifierArray will copy the qualifier bytes.
> ScanDeleteTracker should keep the cell rather than qualifier array.
> {noformat}
> public void add(Cell cell) {
> long timestamp = cell.getTimestamp();
> byte type = cell.getTypeByte();
> if (!hasFamilyStamp || timestamp > familyStamp) {
> if (type == KeyValue.Type.DeleteFamily.getCode()) {
> hasFamilyStamp = true;
> familyStamp = timestamp;
> return;
> } else if (type == KeyValue.Type.DeleteFamilyVersion.getCode()) {
> familyVersionStamps.add(timestamp);
> return;
> }
> if (deleteBuffer != null && type < deleteType) {
> // same column, so ignore less specific delete
> if (CellUtil.matchingQualifier(cell, deleteBuffer, deleteOffset,
> deleteLength)) {
> return;
> }
> }
> // new column, or more general delete type
> deleteBuffer = cell.getQualifierArray();
> deleteOffset = cell.getQualifierOffset();
> deleteLength = cell.getQualifierLength();
> deleteType = type;
> deleteTimestamp = timestamp;
> }
> // missing else is never called.
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17739) BucketCache is inefficient/wasteful/dumb in its bucket allocations
[ https://issues.apache.org/jira/browse/HBASE-17739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898603#comment-15898603 ] chunhui shen commented on HBASE-17739: -- How about making the encoded blocks have similar size? In fact, we did this to increase the ratio of bucket cache space. If any interesting or aggree so , I could open a new issue to upload the patch. > BucketCache is inefficient/wasteful/dumb in its bucket allocations > -- > > Key: HBASE-17739 > URL: https://issues.apache.org/jira/browse/HBASE-17739 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > By default we allocate 14 buckets with sizes from 5K to 513K. If lots of heap > given over to bucketcache and say no allocattions made for a particular > bucket size, this means we have a bunch of the bucketcache that just goes > idle/unused. > For example, say heap is 100G. We'll divide it up among the sizes. If say we > only ever do 5k records, then most of the cache will go unused while the > allocation for 5k objects will see churn. > Here is an old note of [~anoop.hbase]'s' from a conversation on bucket cache > we had offlist that describes the issue: > "By default we have those 14 buckets with size range of 5K to 513K. > All sizes will have one bucket (with size 513*4) each except the > last size.. ie. 513K sized many buckets will be there. If we keep on > writing only same sized blocks, we may loose all in btw sized buckets. > Say we write only 4K sized blocks. We will 1st fill the bucket in 5K > size. There is only one such bucket. Once this is filled, we will try > to grab a complete free bucket from other sizes.. But we can not take > it from 9K... 385K sized ones as there is only ONE bucket for these > sizes. We will take only from 513 size.. There are many in that... > So we will eventually take all the buckets from 513 except the last > one.. Ya it has to keep at least one in evey size.. So we will > loose these much size.. They are of no use." > We should set the size type on the fly as the records come in. > Or better, we should choose record size on the fly. Here is another comment > from [~anoop.hbase]: > "The second is the biggest contributor. Suppose instead of 4K > sized blocks, the user has 2 K sized blocks.. When we write a block to > bucket slot, we will reserve size equal to the allocated size for that block. > So when we write 2K sized blocks (may be actual size a bit more than > 2K ) we will take 5K with each of the block. So u can see that we are > loosing ~3K with every block. Means we are loosing more than half." > He goes on: "If am 100% sure that all my table having 2K HFile block size, I > need to give this config a value 3 * 1024 (Exact 2 K if I give there may be > again problem! That is another story we need to see how we can give > more guarantee for the block size restriction HBASE-15248).. So here also > ~1K loose for every 2K.. So some thing like a 30% loose !!! :-(“" > So, we should figure the record sizes ourselves on the fly. > Anything less has us wasting loads of cache space, nvm inefficiences lost > because of how we serialize base types to cache. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17734) Guard against possibly copying the qualifier in the ScanDeleteTracker
[
https://issues.apache.org/jira/browse/HBASE-17734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17734:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> Guard against possibly copying the qualifier in the ScanDeleteTracker
> -
>
> Key: HBASE-17734
> URL: https://issues.apache.org/jira/browse/HBASE-17734
> Project: HBase
> Issue Type: Improvement
>Reporter: CHIA-PING TSAI
>Assignee: CHIA-PING TSAI
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17734.v0.patch, HBASE-17734.v1.patch,
> HBASE-17734.v2.patch
>
>
> If the input cell is ByteBufferKeyValue, the
> ByteBufferKeyValue#getQualifierArray will copy the qualifier bytes.
> ScanDeleteTracker should keep the cell rather than qualifier array.
> {noformat}
> public void add(Cell cell) {
> long timestamp = cell.getTimestamp();
> byte type = cell.getTypeByte();
> if (!hasFamilyStamp || timestamp > familyStamp) {
> if (type == KeyValue.Type.DeleteFamily.getCode()) {
> hasFamilyStamp = true;
> familyStamp = timestamp;
> return;
> } else if (type == KeyValue.Type.DeleteFamilyVersion.getCode()) {
> familyVersionStamps.add(timestamp);
> return;
> }
> if (deleteBuffer != null && type < deleteType) {
> // same column, so ignore less specific delete
> if (CellUtil.matchingQualifier(cell, deleteBuffer, deleteOffset,
> deleteLength)) {
> return;
> }
> }
> // new column, or more general delete type
> deleteBuffer = cell.getQualifierArray();
> deleteOffset = cell.getQualifierOffset();
> deleteLength = cell.getQualifierLength();
> deleteType = type;
> deleteTimestamp = timestamp;
> }
> // missing else is never called.
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17734) Guard against possibly copying the qualifier in the ScanDeleteTracker
[
https://issues.apache.org/jira/browse/HBASE-17734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898596#comment-15898596
]
Ted Yu commented on HBASE-17734:
Corrected typo in subject when committing: coping -> copying
> Guard against possibly copying the qualifier in the ScanDeleteTracker
> -
>
> Key: HBASE-17734
> URL: https://issues.apache.org/jira/browse/HBASE-17734
> Project: HBase
> Issue Type: Improvement
>Reporter: CHIA-PING TSAI
>Assignee: CHIA-PING TSAI
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17734.v0.patch, HBASE-17734.v1.patch,
> HBASE-17734.v2.patch
>
>
> If the input cell is ByteBufferKeyValue, the
> ByteBufferKeyValue#getQualifierArray will copy the qualifier bytes.
> ScanDeleteTracker should keep the cell rather than qualifier array.
> {noformat}
> public void add(Cell cell) {
> long timestamp = cell.getTimestamp();
> byte type = cell.getTypeByte();
> if (!hasFamilyStamp || timestamp > familyStamp) {
> if (type == KeyValue.Type.DeleteFamily.getCode()) {
> hasFamilyStamp = true;
> familyStamp = timestamp;
> return;
> } else if (type == KeyValue.Type.DeleteFamilyVersion.getCode()) {
> familyVersionStamps.add(timestamp);
> return;
> }
> if (deleteBuffer != null && type < deleteType) {
> // same column, so ignore less specific delete
> if (CellUtil.matchingQualifier(cell, deleteBuffer, deleteOffset,
> deleteLength)) {
> return;
> }
> }
> // new column, or more general delete type
> deleteBuffer = cell.getQualifierArray();
> deleteOffset = cell.getQualifierOffset();
> deleteLength = cell.getQualifierLength();
> deleteType = type;
> deleteTimestamp = timestamp;
> }
> // missing else is never called.
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17717) Incorrect ZK ACL set for HBase superuser
[
https://issues.apache.org/jira/browse/HBASE-17717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898585#comment-15898585
]
Hudson commented on HBASE-17717:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #107 (See
[https://builds.apache.org/job/HBase-1.2-JDK8/107/])
HBASE-17717 Explicitly use "sasl" ACL scheme for hbase superuser (elserj: rev
628490b29e20cb33257d0997b28d14cfd3e9a456)
* (edit)
hbase-client/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKUtil.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
> Incorrect ZK ACL set for HBase superuser
>
>
> Key: HBASE-17717
> URL: https://issues.apache.org/jira/browse/HBASE-17717
> Project: HBase
> Issue Type: Bug
> Components: security, Zookeeper
>Reporter: Shreya Bhat
>Assignee: Josh Elser
> Fix For: 2.0.0, 1.4.0, 1.3.1, 1.1.10, 1.2.6
>
> Attachments: HBASE-17717.001.0.98.patch,
> HBASE-17717.001.branch-1.1.patch, HBASE-17717.001.patch
>
>
> Shreya was doing some testing of a deploy of HBase, verifying that the ZK
> ACLs were actually set as we expect (yay, security).
> She noticed that, in some cases, we were seeing multiple ACLs for the same
> user.
> {noformat}
> 'world,'anyone
> : r
> 'sasl,'hbase
> : cdrwa
> 'sasl,'hbase
> : cdrwa
> {noformat}
> After digging into this (and some insight from the mighty [~enis]), we
> realized that this was happening because of an overridden value for
> {{hbase.superuser}}. However, the ACL value doesn't match what we'd expect to
> see (as hbase.superuser was set to {{cstm-hbase}}).
> After digging into this code, it seems like the {{auth}} ACL scheme in
> ZooKeeper does not work as we expect.
> {code}
> if (superUser != null) {
> acls.add(new ACL(Perms.ALL, new Id("auth", superUser)));
> }
> {code}
> In the above, the {{"auth"}} scheme ignores any provided "subject" in the
> {{Id}} object. It *only* considers the authentication of the current
> connection. As such, our usage of this never actually sets the ACL for the
> superuser correctly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17734) Guard against possibly copying the qualifier in the ScanDeleteTracker
[
https://issues.apache.org/jira/browse/HBASE-17734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898580#comment-15898580
]
CHIA-PING TSAI commented on HBASE-17734:
Thanks for the review. [~yuzhih...@gmail.com] [~anoop.hbase]
> Guard against possibly copying the qualifier in the ScanDeleteTracker
> -
>
> Key: HBASE-17734
> URL: https://issues.apache.org/jira/browse/HBASE-17734
> Project: HBase
> Issue Type: Improvement
>Reporter: CHIA-PING TSAI
>Assignee: CHIA-PING TSAI
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17734.v0.patch, HBASE-17734.v1.patch,
> HBASE-17734.v2.patch
>
>
> If the input cell is ByteBufferKeyValue, the
> ByteBufferKeyValue#getQualifierArray will copy the qualifier bytes.
> ScanDeleteTracker should keep the cell rather than qualifier array.
> {noformat}
> public void add(Cell cell) {
> long timestamp = cell.getTimestamp();
> byte type = cell.getTypeByte();
> if (!hasFamilyStamp || timestamp > familyStamp) {
> if (type == KeyValue.Type.DeleteFamily.getCode()) {
> hasFamilyStamp = true;
> familyStamp = timestamp;
> return;
> } else if (type == KeyValue.Type.DeleteFamilyVersion.getCode()) {
> familyVersionStamps.add(timestamp);
> return;
> }
> if (deleteBuffer != null && type < deleteType) {
> // same column, so ignore less specific delete
> if (CellUtil.matchingQualifier(cell, deleteBuffer, deleteOffset,
> deleteLength)) {
> return;
> }
> }
> // new column, or more general delete type
> deleteBuffer = cell.getQualifierArray();
> deleteOffset = cell.getQualifierOffset();
> deleteLength = cell.getQualifierLength();
> deleteType = type;
> deleteTimestamp = timestamp;
> }
> // missing else is never called.
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17734) Guard against possibly copying the qualifier in the ScanDeleteTracker
[
https://issues.apache.org/jira/browse/HBASE-17734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898576#comment-15898576
]
Anoop Sam John commented on HBASE-17734:
+1
> Guard against possibly copying the qualifier in the ScanDeleteTracker
> -
>
> Key: HBASE-17734
> URL: https://issues.apache.org/jira/browse/HBASE-17734
> Project: HBase
> Issue Type: Improvement
>Reporter: CHIA-PING TSAI
>Assignee: CHIA-PING TSAI
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17734.v0.patch, HBASE-17734.v1.patch,
> HBASE-17734.v2.patch
>
>
> If the input cell is ByteBufferKeyValue, the
> ByteBufferKeyValue#getQualifierArray will copy the qualifier bytes.
> ScanDeleteTracker should keep the cell rather than qualifier array.
> {noformat}
> public void add(Cell cell) {
> long timestamp = cell.getTimestamp();
> byte type = cell.getTypeByte();
> if (!hasFamilyStamp || timestamp > familyStamp) {
> if (type == KeyValue.Type.DeleteFamily.getCode()) {
> hasFamilyStamp = true;
> familyStamp = timestamp;
> return;
> } else if (type == KeyValue.Type.DeleteFamilyVersion.getCode()) {
> familyVersionStamps.add(timestamp);
> return;
> }
> if (deleteBuffer != null && type < deleteType) {
> // same column, so ignore less specific delete
> if (CellUtil.matchingQualifier(cell, deleteBuffer, deleteOffset,
> deleteLength)) {
> return;
> }
> }
> // new column, or more general delete type
> deleteBuffer = cell.getQualifierArray();
> deleteOffset = cell.getQualifierOffset();
> deleteLength = cell.getQualifierLength();
> deleteType = type;
> deleteTimestamp = timestamp;
> }
> // missing else is never called.
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-15484) Correct the semantic of batch and partial
[ https://issues.apache.org/jira/browse/HBASE-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-15484: -- Affects Version/s: (was: 1.1.3) (was: 1.2.0) 1.4.0 2.0.0 Fix Version/s: 1.4.0 Component/s: scan Client > Correct the semantic of batch and partial > - > > Key: HBASE-15484 > URL: https://issues.apache.org/jira/browse/HBASE-15484 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Affects Versions: 2.0.0, 1.4.0 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15484.branch-1.v01.patch, HBASE-15484.v05.patch, > HBASE-15484.v06.patch, HBASE-15484-v1.patch, HBASE-15484-v2.patch, > HBASE-15484-v3.patch, HBASE-15484-v4.patch > > > Follow-up to HBASE-15325, as discussed, the meaning of setBatch and > setAllowPartialResults should not be same. We should not regard setBatch as > setAllowPartialResults. > Now we deprecated isPartial and use mayHaveMoreCellsInRow. If it returns > false, current Result must be the last one of this row. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17740) Correct the semantic of batch and partial for async client
Duo Zhang created HBASE-17740: - Summary: Correct the semantic of batch and partial for async client Key: HBASE-17740 URL: https://issues.apache.org/jira/browse/HBASE-17740 Project: HBase Issue Type: Sub-task Components: asyncclient, Client, scan Affects Versions: 2.0.0 Reporter: Duo Zhang Fix For: 2.0.0 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15484) Correct the semantic of batch and partial
[ https://issues.apache.org/jira/browse/HBASE-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898567#comment-15898567 ] Duo Zhang commented on HBASE-15484: --- +1. Can fix the javadoc issues on commit. > Correct the semantic of batch and partial > - > > Key: HBASE-15484 > URL: https://issues.apache.org/jira/browse/HBASE-15484 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-15484.branch-1.v01.patch, HBASE-15484.v05.patch, > HBASE-15484.v06.patch, HBASE-15484-v1.patch, HBASE-15484-v2.patch, > HBASE-15484-v3.patch, HBASE-15484-v4.patch > > > Follow-up to HBASE-15325, as discussed, the meaning of setBatch and > setAllowPartialResults should not be same. We should not regard setBatch as > setAllowPartialResults. > Now we deprecated isPartial and use mayHaveMoreCellsInRow. If it returns > false, current Result must be the last one of this row. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17532) Replace explicit type with diamond operator
[ https://issues.apache.org/jira/browse/HBASE-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898564#comment-15898564 ] stack commented on HBASE-17532: --- Patch had rotted. I update it [~Jan Hentschel] Lets get your work committed. > Replace explicit type with diamond operator > --- > > Key: HBASE-17532 > URL: https://issues.apache.org/jira/browse/HBASE-17532 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17532.master.001.patch, > HBASE-17532.master.002.patch > > > Because HBase uses Java 8 explicit types can be replaced with the diamond > operator. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17532) Replace explicit type with diamond operator
[ https://issues.apache.org/jira/browse/HBASE-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17532: -- Attachment: HBASE-17532.master.002.patch > Replace explicit type with diamond operator > --- > > Key: HBASE-17532 > URL: https://issues.apache.org/jira/browse/HBASE-17532 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17532.master.001.patch, > HBASE-17532.master.002.patch > > > Because HBase uses Java 8 explicit types can be replaced with the diamond > operator. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17712) Remove/Simplify the logic of RegionScannerImpl.handleFileNotFound
[ https://issues.apache.org/jira/browse/HBASE-17712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898535#comment-15898535 ] Duo Zhang commented on HBASE-17712: --- What do you think of the new approach sir? [~stack] Thanks. > Remove/Simplify the logic of RegionScannerImpl.handleFileNotFound > - > > Key: HBASE-17712 > URL: https://issues.apache.org/jira/browse/HBASE-17712 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17712.patch, HBASE-17712-ut.patch, > HBASE-17712-v1.patch, HBASE-17712-v2.patch, HBASE-17712-v3.patch > > > It is introduced in HBASE-13651 and the logic became much more complicated > after HBASE-16304 due to a dead lock issue. It is really tough as sequence id > is involved in and the method we called is used to serve secondary replica > originally which does not handle write. > In fact, in 1.x release, the problem described in HBASE-13651 is gone. Now we > will write a compaction marker to WAL before deleting the compacted files. We > can only consider a RS as dead after its WAL files are all closed so if the > region has already been reassigned the compaction will fail as we can not > write out the compaction marker. > So theoretically, if we still hit FileNotFound exception, it should be a > critical bug which means we may loss data. I do not think it is a good idea > to just eat the exception and refresh store files. Or even if we want to do > this, we can just refresh store files without dropping memstore contents. > This will also simplify the logic a lot. > Suggestions are welcomed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17728) Create separate build target for Configuration classes
[ https://issues.apache.org/jira/browse/HBASE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17728: --- Attachment: 17728.v4.txt Patch v4 adds std::call_once to isSecurityEnabled() > Create separate build target for Configuration classes > -- > > Key: HBASE-17728 > URL: https://issues.apache.org/jira/browse/HBASE-17728 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 17728.v1.txt, 17728.v2.txt, 17728.v3.txt, 17728.v4.txt > > > User is in security module. > When User::isSecurityEnabled() is added, we need to query Configuration for > security setting. > However, this introduces a circular build dependency: > BUILD FAILED: Cycle found: //connection:connection -> //security:security -> > //core:core -> //connection:connection > This issue is to create separate build target for Configuration which is > depended upon by both core and security modules. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17728) Create separate build target for Configuration classes
[ https://issues.apache.org/jira/browse/HBASE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17728: --- Description: User is in security module. When User::isSecurityEnabled() is added, we need to query Configuration for security setting. However, this introduces a circular build dependency: BUILD FAILED: Cycle found: //connection:connection -> //security:security -> //core:core -> //connection:connection This issue is to create separate build target for Configuration which is depended upon by both core and security modules. was: User is in security module. When User::isSecurityEnabled() is added, we need to query Configuration for security setting. However, this introduces a circular build dependency: BUILD FAILED: Cycle found: //connection:connection -> //security:security -> //core:core -> //connection:connection This issue is to separate Configuration into common module which is depended upon by both core and connection modules. > Create separate build target for Configuration classes > -- > > Key: HBASE-17728 > URL: https://issues.apache.org/jira/browse/HBASE-17728 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 17728.v1.txt, 17728.v2.txt, 17728.v3.txt > > > User is in security module. > When User::isSecurityEnabled() is added, we need to query Configuration for > security setting. > However, this introduces a circular build dependency: > BUILD FAILED: Cycle found: //connection:connection -> //security:security -> > //core:core -> //connection:connection > This issue is to create separate build target for Configuration which is > depended upon by both core and security modules. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17728) Create separate build target for Configuration classes
[ https://issues.apache.org/jira/browse/HBASE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17728: --- Summary: Create separate build target for Configuration classes (was: Separate Configuration into common module) > Create separate build target for Configuration classes > -- > > Key: HBASE-17728 > URL: https://issues.apache.org/jira/browse/HBASE-17728 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 17728.v1.txt, 17728.v2.txt, 17728.v3.txt > > > User is in security module. > When User::isSecurityEnabled() is added, we need to query Configuration for > security setting. > However, this introduces a circular build dependency: > BUILD FAILED: Cycle found: //connection:connection -> //security:security -> > //core:core -> //connection:connection > This issue is to separate Configuration into common module which is depended > upon by both core and connection modules. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17002) [Master] Report quotas and active violations on Master UI and JMX metrics
[
https://issues.apache.org/jira/browse/HBASE-17002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898459#comment-15898459
]
Hadoop QA commented on HBASE-17002:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 29s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
46s {color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 30s
{color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 15m
23s {color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
46s {color} | {color:green} HBASE-16961 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 50s
{color} | {color:red} hbase-protocol-shaded in HBASE-16961 has 24 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} HBASE-16961 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 15m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 55s
{color} | {color:red} The patch causes 306 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 51s
{color} | {color:red} The patch causes 306 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 46s
{color} | {color:red} The patch causes 306 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 43s
{color} | {color:red} The patch causes 306 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 42s
{color} | {color:red} The patch causes 306 errors with Hadoop v2.5.2. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 11s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 82m 21s
{color} | {color:green} hbase-server in the patch passed. {color} |
|
[jira] [Commented] (HBASE-17718) Difference between RS's servername and its ephemeral node cause SSH stop working
[
https://issues.apache.org/jira/browse/HBASE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898452#comment-15898452
]
Hadoop QA commented on HBASE-17718:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 39s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
23s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 22s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 52s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 56s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 120m 47s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestRSKilledWhenInitializing |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:e01ee2f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856354/HBASE-17718.branch-1.001.patch
|
| JIRA Issue | HBASE-17718 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 5903e64f5e01 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision |
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898444#comment-15898444 ] stack commented on HBASE-14123: --- The release note looks good. Suggest you cut it down so that it is about this commit. As is, most of release note is about what is coming in follow-ons which will confuse the user trying to figure what is contained herein. Just make it about what this adds. Bulk up the limitations referring to the follow-on JIRAs (ok to point to follow-on work and the fixed before hbase-2). When done, suggest you put this commit up for a vote. Paste in the edited release note into the VOTE so folks know what they are voting on. Are the findbugs and failed unit tests yours? Thanks Vladimir > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Fix For: HBASE-7912 > > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v20.txt, 14123-master.v21.txt, > 14123-master.v24.txt, 14123-master.v25.txt, 14123-master.v27.txt, > 14123-master.v28.txt, 14123-master.v29.full.txt, 14123-master.v2.txt, > 14123-master.v30.txt, 14123-master.v31.txt, 14123-master.v32.txt, > 14123-master.v33.txt, 14123-master.v34.txt, 14123-master.v35.txt, > 14123-master.v36.txt, 14123-master.v37.txt, 14123-master.v38.txt, > 14123.master.v39.patch, 14123-master.v3.txt, 14123.master.v40.patch, > 14123.master.v41.patch, 14123.master.v42.patch, 14123.master.v44.patch, > 14123.master.v45.patch, 14123.master.v46.patch, 14123.master.v48.patch, > 14123.master.v49.patch, 14123.master.v50.patch, 14123.master.v51.patch, > 14123.master.v52.patch, 14123.master.v54.patch, 14123.master.v56.patch, > 14123.master.v57.patch, 14123.master.v58.patch, 14123.master.v59.patch, > 14123-master.v5.txt, 14123.master.v60.patch, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > Backup-restoreinHBase2.0 (1).pdf, Backup-restoreinHBase2.0 (3).pdf, > Backup-restoreinHBase2.0.pdf, HBASE-14123-for-7912-v1.patch, > HBASE-14123-for-7912-v6.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v1.patch, HBASE-14123-v2.patch, > HBASE-14123-v3.patch, HBASE-14123-v4.patch, HBASE-14123-v5.patch, > HBASE-14123-v6.patch, HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17718) Difference between RS's servername and its ephemeral node cause SSH stop working
[
https://issues.apache.org/jira/browse/HBASE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898440#comment-15898440
]
Hudson commented on HBASE-17718:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2625 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2625/])
HBASE-17718 Difference between RS's servername and its ephemeral node (stack:
rev 7fa7156f2cfbc6a5c9a50739d0ea4a3d5ce4c6ce)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterProcedureEvents.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSKilledWhenInitializing.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestServerCrashProcedure.java
> Difference between RS's servername and its ephemeral node cause SSH stop
> working
>
>
> Key: HBASE-17718
> URL: https://issues.apache.org/jira/browse/HBASE-17718
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.2.4, 1.1.8
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17718.branch-1.001.patch,
> HBASE-17718.master.001.patch, HBASE-17718.master.002.patch,
> HBASE-17718.master.003.patch
>
>
> After HBASE-9593, RS put up an ephemeral node in ZK before reporting for
> duty. But if the hosts config (/etc/hosts) is different between master and
> RS, RS's serverName can be different from the one stored the ephemeral zk
> node. The email metioned in HBASE-13753
> (http://mail-archives.apache.org/mod_mbox/hbase-user/201505.mbox/%3CCANZDn9ueFEEuZMx=pZdmtLsdGLyZz=rrm1N6EQvLswYc1z-H=g...@mail.gmail.com%3E)
> is exactly what happened in our production env.
> But what the email didn't point out is that the difference between serverName
> in RS and zk node can cause SSH stop to work. as we can see from the code in
> {{RegionServerTracker}}
> {code}
> @Override
> public void nodeDeleted(String path) {
> if (path.startsWith(watcher.rsZNode)) {
> String serverName = ZKUtil.getNodeName(path);
> LOG.info("RegionServer ephemeral node deleted, processing expiration ["
> +
> serverName + "]");
> ServerName sn = ServerName.parseServerName(serverName);
> if (!serverManager.isServerOnline(sn)) {
> LOG.warn(serverName.toString() + " is not online or isn't known to
> the master."+
> "The latter could be caused by a DNS misconfiguration.");
> return;
> }
> remove(sn);
> this.serverManager.expireServer(sn);
> }
> }
> {code}
> The server will not be processed by SSH/ServerCrashProcedure. The regions on
> this server will not been assigned again until master restart or failover.
> I know HBASE-9593 was to fix the issue if RS report to duty and crashed
> before it can put up a zk node. It is a very rare case(And controllable， just
> fix the bug making rs to crash). But The issue I metioned can happened more
> often(and uncontrollable, can't be fixed in HBase, due to DNS, hosts config,
> etc.) and have more severe consequence.
> So here I offer some solutions to discuss:
> 1. Revert HBASE-9593 from all branches, Andrew Purtell has reverted it in
> branch-0.98
> 2. Abort RS if master return a different name, otherwise SSH can't work
> properly
> 3. Master accepts whatever servername reported by RS and don't change it.
> 4.correct the zk node if master return another name( idea from Ted Yu)
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17680) Run mini cluster through JNI in tests
[ https://issues.apache.org/jira/browse/HBASE-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-17680: -- Fix Version/s: HBASE-14850 > Run mini cluster through JNI in tests > - > > Key: HBASE-17680 > URL: https://issues.apache.org/jira/browse/HBASE-17680 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: HBASE-14850 > > Attachments: 17680.v14.txt, 17680.v17.txt, 17680.v18.txt, > 17680.v1.txt, 17680.v20.txt, 17680.v22.txt, 17680.v23.txt, 17680.v26.txt, > 17680.v27.txt, 17680.v28.txt, 17680.v29.txt, 17680.v30.txt, 17680.v31.txt, > 17680.v33.txt, 17680.v3.txt, 17680.v8.txt > > > Currently tests start local hbase cluster through hbase shell. > There is less control over the configuration of the local cluster this way. > This issue would replace hbase shell with JNI interface to mini cluster. > We would have full control over the cluster behavior. > Thanks to [~devaraj] who started this initiative. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17698) ReplicationEndpoint choosing sinks
[ https://issues.apache.org/jira/browse/HBASE-17698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karan Mehta updated HBASE-17698: Status: Patch Available (was: Open) > ReplicationEndpoint choosing sinks > -- > > Key: HBASE-17698 > URL: https://issues.apache.org/jira/browse/HBASE-17698 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0 >Reporter: churro morales >Assignee: Karan Mehta > Attachments: HBASE-17698.patch > > > The only time we choose new sinks is when we have a ConnectException, but we > have encountered other exceptions where there is a problem contacting a > particular sink and replication gets backed up for any sources that try that > sink > HBASE-17675 occurred when there was a bad keytab refresh and the source was > stuck. > Another issue we recently had was a bad drive controller on the sink side and > replication was stuck again. > Is there any reason not to choose new sinks anytime we have a > RemoteException? I can understand TableNotFound we don't have to choose new > sinks, but for all other cases this seems like the safest approach. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17698) ReplicationEndpoint choosing sinks
[ https://issues.apache.org/jira/browse/HBASE-17698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karan Mehta updated HBASE-17698: Attachment: HBASE-17698.patch > ReplicationEndpoint choosing sinks > -- > > Key: HBASE-17698 > URL: https://issues.apache.org/jira/browse/HBASE-17698 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0 >Reporter: churro morales >Assignee: Karan Mehta > Attachments: HBASE-17698.patch > > > The only time we choose new sinks is when we have a ConnectException, but we > have encountered other exceptions where there is a problem contacting a > particular sink and replication gets backed up for any sources that try that > sink > HBASE-17675 occurred when there was a bad keytab refresh and the source was > stuck. > Another issue we recently had was a bad drive controller on the sink side and > replication was stuck again. > Is there any reason not to choose new sinks anytime we have a > RemoteException? I can understand TableNotFound we don't have to choose new > sinks, but for all other cases this seems like the safest approach. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17739) BucketCache is inefficient/wasteful/dumb in its bucket allocations
stack created HBASE-17739: - Summary: BucketCache is inefficient/wasteful/dumb in its bucket allocations Key: HBASE-17739 URL: https://issues.apache.org/jira/browse/HBASE-17739 Project: HBase Issue Type: Sub-task Components: BucketCache Reporter: stack By default we allocate 14 buckets with sizes from 5K to 513K. If lots of heap given over to bucketcache and say no allocattions made for a particular bucket size, this means we have a bunch of the bucketcache that just goes idle/unused. For example, say heap is 100G. We'll divide it up among the sizes. If say we only ever do 5k records, then most of the cache will go unused while the allocation for 5k objects will see churn. Here is an old note of [~anoop.hbase]'s' from a conversation on bucket cache we had offlist that describes the issue: "By default we have those 14 buckets with size range of 5K to 513K. All sizes will have one bucket (with size 513*4) each except the last size.. ie. 513K sized many buckets will be there. If we keep on writing only same sized blocks, we may loose all in btw sized buckets. Say we write only 4K sized blocks. We will 1st fill the bucket in 5K size. There is only one such bucket. Once this is filled, we will try to grab a complete free bucket from other sizes.. But we can not take it from 9K... 385K sized ones as there is only ONE bucket for these sizes. We will take only from 513 size.. There are many in that... So we will eventually take all the buckets from 513 except the last one.. Ya it has to keep at least one in evey size.. So we will loose these much size.. They are of no use." We should set the size type on the fly as the records come in. Or better, we should choose record size on the fly. Here is another comment from [~anoop.hbase]: "The second is the biggest contributor. Suppose instead of 4K sized blocks, the user has 2 K sized blocks.. When we write a block to bucket slot, we will reserve size equal to the allocated size for that block. So when we write 2K sized blocks (may be actual size a bit more than 2K ) we will take 5K with each of the block. So u can see that we are loosing ~3K with every block. Means we are loosing more than half." He goes on: "If am 100% sure that all my table having 2K HFile block size, I need to give this config a value 3 * 1024 (Exact 2 K if I give there may be again problem! That is another story we need to see how we can give more guarantee for the block size restriction HBASE-15248).. So here also ~1K loose for every 2K.. So some thing like a 30% loose !!! :-(“" So, we should figure the record sizes ourselves on the fly. Anything less has us wasting loads of cache space, nvm inefficiences lost because of how we serialize base types to cache. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17738) BucketCache startup is slow
stack created HBASE-17738: - Summary: BucketCache startup is slow Key: HBASE-17738 URL: https://issues.apache.org/jira/browse/HBASE-17738 Project: HBase Issue Type: Sub-task Components: BucketCache Reporter: stack If you set bucketcache size at 64G say and then start hbase, it takes a long time. Can we do the allocations in parallel and not inline with the server startup? Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17728) Separate Configuration into common module
[ https://issues.apache.org/jira/browse/HBASE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17728: --- Attachment: 17728.v3.txt Patch v3 creates a build target, conf, inside core/BUCK core:core depends on core:conf The following command passes: buck test //core:filter-test make also passes. I can refine isSecurityEnabled() if reviewers think this approach is good. > Separate Configuration into common module > - > > Key: HBASE-17728 > URL: https://issues.apache.org/jira/browse/HBASE-17728 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 17728.v1.txt, 17728.v2.txt, 17728.v3.txt > > > User is in security module. > When User::isSecurityEnabled() is added, we need to query Configuration for > security setting. > However, this introduces a circular build dependency: > BUILD FAILED: Cycle found: //connection:connection -> //security:security -> > //core:core -> //connection:connection > This issue is to separate Configuration into common module which is depended > upon by both core and connection modules. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15248) BLOCKSIZE 4k should result in 4096 bytes on disk; i.e. fit inside a BucketCache 'block' of 4k
[
https://issues.apache.org/jira/browse/HBASE-15248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898383#comment-15898383
]
stack commented on HBASE-15248:
---
Added this note to BLOCKSIZE:
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
@@ -103,7 +103,10 @@ public class HColumnDescriptor implements
Comparable {
/**
* Size of storefile/hfile 'blocks'. Default is {@link #DEFAULT_BLOCKSIZE}.
* Use smaller block sizes for faster random-access at expense of larger
- * indices (more memory consumption).
+ * indices (more memory consumption). Note that this is a soft limit and that
+ * blocks have overhead (metadata, CRCs) so blocks will tend to be the size
+ * specified here and then some; i.e. don't expect that setting BLOCKSIZE=4k
+ * means hbase data will align with an SSDs 4k page accesses (TODO).
*/
public static final String BLOCKSIZE = "BLOCKSIZE";
> BLOCKSIZE 4k should result in 4096 bytes on disk; i.e. fit inside a
> BucketCache 'block' of 4k
> -
>
> Key: HBASE-15248
> URL: https://issues.apache.org/jira/browse/HBASE-15248
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: stack
>
> Chatting w/ a gentleman named Daniel Pol who is messing w/ bucketcache, he
> wants blocks to be the size specified in the configuration and no bigger. His
> hardware set ups fetches pages of 4k and so a block that has 4k of payload
> but has then a header and the header of the next block (which helps figure
> whats next when scanning) ends up being 4203 bytes or something, and this
> then then translates into two seeks per block fetch.
> This issue is about what it would take to stay inside our configured size
> boundary writing out blocks.
> If not possible, give back better signal on what to do so you could fit
> inside a particular constraint.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-15457) [performance] Save-a-seek; hint HFileReader when Scan is a "Get" Scan
[ https://issues.apache.org/jira/browse/HBASE-15457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15457: -- Component/s: Performance HFile Issue Type: Improvement (was: Bug) > [performance] Save-a-seek; hint HFileReader when Scan is a "Get" Scan > - > > Key: HBASE-15457 > URL: https://issues.apache.org/jira/browse/HBASE-15457 > Project: HBase > Issue Type: Improvement > Components: HFile, Performance >Reporter: stack > > Have the Scan hint the lower-level Reader when its a 'Get' Scan. Reader is > currently doing checks for EOF and when time to load next block on each next > invocation. Seems easy enough to return null/end-of-scan if a get-scan and > the next block is a different row. > Prompted by @daniel pol questions/suggestions over on HBASE-15392; see > towards the end. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17576) [C++] Implement request retry mechanism over RPC for Multi calls.
[
https://issues.apache.org/jira/browse/HBASE-17576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898352#comment-15898352
]
Enis Soztutar commented on HBASE-17576:
---
- Some of these should be {{int32_t}} or {{size_t}}:
{code}
+ Action(std::shared_ptr action, int original_index)
+int hbase::MultiAction::Size() const {
+ int size = 0;
...
{code}
probably you were not able to run {{make lint}} yet. We can do that close to
commit.
- Remove these:
{code}
+ // std::string row_ = "";
+ // const std::string& Row() const;
+// const std::string ::Row() const { return row_; }
{code}
and also other commented out code.
- We should be using only 1 way to name files. In the current code base, there
are both {{foo-bar.cc}} and {{foo_bar.cc}} used. Let's just stick to using
{{foo-bar.cc}} for all of the new files. For example, {{region_result.h}}
should be {{region-result.h}}, etc. I'll fix this in a different jira in the
existing code base.
- Returning a {{const shared_ptr &}} does not make sense to me. Maybe you
should return just {{const Row&}} here:
{code}
+ const std::shared_ptr () { return action_; }
{code}
- We do not need this class: {{AsyncRpcRetryingBatchCallerFactory}}. Instead
the code should go inside the {{AsyncRpcRetryingCallerFactory}} class. We want
{{AsyncRpcRetryingCallerFactory::Batch()}} call similar to the {{::Single()}}
call.
- In class {{BatchCallerBuilder}}, you can use the C++ conventions to name the
methods like SetActions, SetTable, etc like actions() tables(), etc. Look at
the {{SingleRequestCallerBuilder}}.
To be continued.
> [C++] Implement request retry mechanism over RPC for Multi calls.
> -
>
> Key: HBASE-17576
> URL: https://issues.apache.org/jira/browse/HBASE-17576
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sudeep Sunthankar
>Assignee: Sudeep Sunthankar
> Attachments: HBASE-17576.HBASE-14850.v1.patch,
> HBASE-17576.HBASE-14850.v2.patch, HBASE-17576.HBASE-14850.v3.patch,
> HBASE-17576.HBASE-14850.v4.patch, HBASE-17576.HBASE-14850.v5.patch
>
>
> This work is based on top of HBASE-17465. Multi Calls will be based on this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17728) Separate Configuration into common module
[ https://issues.apache.org/jira/browse/HBASE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17728: --- Attachment: 17728.v2.txt Patch v2 is rebased after HBASE-17680 came in. > Separate Configuration into common module > - > > Key: HBASE-17728 > URL: https://issues.apache.org/jira/browse/HBASE-17728 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 17728.v1.txt, 17728.v2.txt > > > User is in security module. > When User::isSecurityEnabled() is added, we need to query Configuration for > security setting. > However, this introduces a circular build dependency: > BUILD FAILED: Cycle found: //connection:connection -> //security:security -> > //core:core -> //connection:connection > This issue is to separate Configuration into common module which is depended > upon by both core and connection modules. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-17680) Run mini cluster through JNI in tests
[ https://issues.apache.org/jira/browse/HBASE-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-17680. Resolution: Fixed Hadoop Flags: Reviewed Thanks for the review, Enis. > Run mini cluster through JNI in tests > - > > Key: HBASE-17680 > URL: https://issues.apache.org/jira/browse/HBASE-17680 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 17680.v14.txt, 17680.v17.txt, 17680.v18.txt, > 17680.v1.txt, 17680.v20.txt, 17680.v22.txt, 17680.v23.txt, 17680.v26.txt, > 17680.v27.txt, 17680.v28.txt, 17680.v29.txt, 17680.v30.txt, 17680.v31.txt, > 17680.v33.txt, 17680.v3.txt, 17680.v8.txt > > > Currently tests start local hbase cluster through hbase shell. > There is less control over the configuration of the local cluster this way. > This issue would replace hbase shell with JNI interface to mini cluster. > We would have full control over the cluster behavior. > Thanks to [~devaraj] who started this initiative. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17680) Run mini cluster through JNI in tests
[
https://issues.apache.org/jira/browse/HBASE-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898321#comment-15898321
]
Enis Soztutar commented on HBASE-17680:
---
Remove this:
{code}
- hbase::Configuration conf;
+ //hbase::Configuration conf;
{code}
Please commit to branch afterwards. Thanks.
> Run mini cluster through JNI in tests
> -
>
> Key: HBASE-17680
> URL: https://issues.apache.org/jira/browse/HBASE-17680
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 17680.v14.txt, 17680.v17.txt, 17680.v18.txt,
> 17680.v1.txt, 17680.v20.txt, 17680.v22.txt, 17680.v23.txt, 17680.v26.txt,
> 17680.v27.txt, 17680.v28.txt, 17680.v29.txt, 17680.v30.txt, 17680.v31.txt,
> 17680.v33.txt, 17680.v3.txt, 17680.v8.txt
>
>
> Currently tests start local hbase cluster through hbase shell.
> There is less control over the configuration of the local cluster this way.
> This issue would replace hbase shell with JNI interface to mini cluster.
> We would have full control over the cluster behavior.
> Thanks to [~devaraj] who started this initiative.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17680) Run mini cluster through JNI in tests
[ https://issues.apache.org/jira/browse/HBASE-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17680: --- Attachment: 17680.v33.txt > Run mini cluster through JNI in tests > - > > Key: HBASE-17680 > URL: https://issues.apache.org/jira/browse/HBASE-17680 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 17680.v14.txt, 17680.v17.txt, 17680.v18.txt, > 17680.v1.txt, 17680.v20.txt, 17680.v22.txt, 17680.v23.txt, 17680.v26.txt, > 17680.v27.txt, 17680.v28.txt, 17680.v29.txt, 17680.v30.txt, 17680.v31.txt, > 17680.v33.txt, 17680.v3.txt, 17680.v8.txt > > > Currently tests start local hbase cluster through hbase shell. > There is less control over the configuration of the local cluster this way. > This issue would replace hbase shell with JNI interface to mini cluster. > We would have full control over the cluster behavior. > Thanks to [~devaraj] who started this initiative. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17680) Run mini cluster through JNI in tests
[ https://issues.apache.org/jira/browse/HBASE-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17680: --- Attachment: (was: 17680.v33.txt) > Run mini cluster through JNI in tests > - > > Key: HBASE-17680 > URL: https://issues.apache.org/jira/browse/HBASE-17680 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 17680.v14.txt, 17680.v17.txt, 17680.v18.txt, > 17680.v1.txt, 17680.v20.txt, 17680.v22.txt, 17680.v23.txt, 17680.v26.txt, > 17680.v27.txt, 17680.v28.txt, 17680.v29.txt, 17680.v30.txt, 17680.v31.txt, > 17680.v33.txt, 17680.v3.txt, 17680.v8.txt > > > Currently tests start local hbase cluster through hbase shell. > There is less control over the configuration of the local cluster this way. > This issue would replace hbase shell with JNI interface to mini cluster. > We would have full control over the cluster behavior. > Thanks to [~devaraj] who started this initiative. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17717) Incorrect ZK ACL set for HBase superuser
[
https://issues.apache.org/jira/browse/HBASE-17717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898313#comment-15898313
]
Hudson commented on HBASE-17717:
SUCCESS: Integrated in Jenkins build HBase-1.2-IT #845 (See
[https://builds.apache.org/job/HBase-1.2-IT/845/])
HBASE-17717 Explicitly use "sasl" ACL scheme for hbase superuser (elserj: rev
628490b29e20cb33257d0997b28d14cfd3e9a456)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
* (edit)
hbase-client/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKUtil.java
> Incorrect ZK ACL set for HBase superuser
>
>
> Key: HBASE-17717
> URL: https://issues.apache.org/jira/browse/HBASE-17717
> Project: HBase
> Issue Type: Bug
> Components: security, Zookeeper
>Reporter: Shreya Bhat
>Assignee: Josh Elser
> Fix For: 2.0.0, 1.4.0, 1.3.1, 1.1.10, 1.2.6
>
> Attachments: HBASE-17717.001.0.98.patch,
> HBASE-17717.001.branch-1.1.patch, HBASE-17717.001.patch
>
>
> Shreya was doing some testing of a deploy of HBase, verifying that the ZK
> ACLs were actually set as we expect (yay, security).
> She noticed that, in some cases, we were seeing multiple ACLs for the same
> user.
> {noformat}
> 'world,'anyone
> : r
> 'sasl,'hbase
> : cdrwa
> 'sasl,'hbase
> : cdrwa
> {noformat}
> After digging into this (and some insight from the mighty [~enis]), we
> realized that this was happening because of an overridden value for
> {{hbase.superuser}}. However, the ACL value doesn't match what we'd expect to
> see (as hbase.superuser was set to {{cstm-hbase}}).
> After digging into this code, it seems like the {{auth}} ACL scheme in
> ZooKeeper does not work as we expect.
> {code}
> if (superUser != null) {
> acls.add(new ACL(Perms.ALL, new Id("auth", superUser)));
> }
> {code}
> In the above, the {{"auth"}} scheme ignores any provided "subject" in the
> {{Id}} object. It *only* considers the authentication of the current
> connection. As such, our usage of this never actually sets the ACL for the
> superuser correctly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17737) Thrift2 proxy should support scan timeRange per column family
[
https://issues.apache.org/jira/browse/HBASE-17737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898306#comment-15898306
]
Ted Yu commented on HBASE-17737:
Wrap long line:
{code}
+ out.setColumnFamilyTimeRange(Bytes.toBytes(entry.getKey()),
entry.getValue().getMinStamp(), entry.getValue().getMaxStamp());
{code}
Can you add a test for the enhancement ?
Fill out release note as well.
> Thrift2 proxy should support scan timeRange per column family
> -
>
> Key: HBASE-17737
> URL: https://issues.apache.org/jira/browse/HBASE-17737
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Yechao Chen
>Assignee: Yechao Chen
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17737.patch
>
>
> see discussion in HBASE-14872
> and HBASE-14355
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17568) Expire region reports in the HMaster
[
https://issues.apache.org/jira/browse/HBASE-17568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898299#comment-15898299
]
Hadoop QA commented on HBASE-17568:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
8s {color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s {color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 3s
{color} | {color:red} The patch causes 306 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 6s
{color} | {color:red} The patch causes 306 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 9s
{color} | {color:red} The patch causes 306 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 11s
{color} | {color:red} The patch causes 306 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 12s
{color} | {color:red} The patch causes 306 errors with Hadoop v2.5.2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 47s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 137m 19s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.client.TestAsyncNonMetaRegionLocatorConcurrenyLimit |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856343/HBASE-17568.003.HBASE-16961.patch
|
| JIRA Issue | HBASE-17568 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 51f320de2d73 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | HBASE-16961 / cc14ccd |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/5969/artifact/patchprocess/whitespace-tabs.txt
[jira] [Commented] (HBASE-17734) Guard against possibly copying the qualifier in the ScanDeleteTracker
[
https://issues.apache.org/jira/browse/HBASE-17734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898300#comment-15898300
]
Ted Yu commented on HBASE-17734:
+1
> Guard against possibly copying the qualifier in the ScanDeleteTracker
> -
>
> Key: HBASE-17734
> URL: https://issues.apache.org/jira/browse/HBASE-17734
> Project: HBase
> Issue Type: Improvement
>Reporter: CHIA-PING TSAI
>Assignee: CHIA-PING TSAI
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17734.v0.patch, HBASE-17734.v1.patch,
> HBASE-17734.v2.patch
>
>
> If the input cell is ByteBufferKeyValue, the
> ByteBufferKeyValue#getQualifierArray will copy the qualifier bytes.
> ScanDeleteTracker should keep the cell rather than qualifier array.
> {noformat}
> public void add(Cell cell) {
> long timestamp = cell.getTimestamp();
> byte type = cell.getTypeByte();
> if (!hasFamilyStamp || timestamp > familyStamp) {
> if (type == KeyValue.Type.DeleteFamily.getCode()) {
> hasFamilyStamp = true;
> familyStamp = timestamp;
> return;
> } else if (type == KeyValue.Type.DeleteFamilyVersion.getCode()) {
> familyVersionStamps.add(timestamp);
> return;
> }
> if (deleteBuffer != null && type < deleteType) {
> // same column, so ignore less specific delete
> if (CellUtil.matchingQualifier(cell, deleteBuffer, deleteOffset,
> deleteLength)) {
> return;
> }
> }
> // new column, or more general delete type
> deleteBuffer = cell.getQualifierArray();
> deleteOffset = cell.getQualifierOffset();
> deleteLength = cell.getQualifierLength();
> deleteType = type;
> deleteTimestamp = timestamp;
> }
> // missing else is never called.
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-15484) Correct the semantic of batch and partial
[ https://issues.apache.org/jira/browse/HBASE-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898289#comment-15898289 ] Ted Yu commented on HBASE-15484: +1 See if the javadoc warning is from your patch. > Correct the semantic of batch and partial > - > > Key: HBASE-15484 > URL: https://issues.apache.org/jira/browse/HBASE-15484 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-15484.branch-1.v01.patch, HBASE-15484.v05.patch, > HBASE-15484.v06.patch, HBASE-15484-v1.patch, HBASE-15484-v2.patch, > HBASE-15484-v3.patch, HBASE-15484-v4.patch > > > Follow-up to HBASE-15325, as discussed, the meaning of setBatch and > setAllowPartialResults should not be same. We should not regard setBatch as > setAllowPartialResults. > Now we deprecated isPartial and use mayHaveMoreCellsInRow. If it returns > false, current Result must be the last one of this row. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17680) Run mini cluster through JNI in tests
[ https://issues.apache.org/jira/browse/HBASE-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17680: --- Attachment: 17680.v33.txt Patch v33 addresses review comments above. All tests pass. > Run mini cluster through JNI in tests > - > > Key: HBASE-17680 > URL: https://issues.apache.org/jira/browse/HBASE-17680 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 17680.v14.txt, 17680.v17.txt, 17680.v18.txt, > 17680.v1.txt, 17680.v20.txt, 17680.v22.txt, 17680.v23.txt, 17680.v26.txt, > 17680.v27.txt, 17680.v28.txt, 17680.v29.txt, 17680.v30.txt, 17680.v31.txt, > 17680.v33.txt, 17680.v3.txt, 17680.v8.txt > > > Currently tests start local hbase cluster through hbase shell. > There is less control over the configuration of the local cluster this way. > This issue would replace hbase shell with JNI interface to mini cluster. > We would have full control over the cluster behavior. > Thanks to [~devaraj] who started this initiative. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17718) Difference between RS's servername and its ephemeral node cause SSH stop working
[
https://issues.apache.org/jira/browse/HBASE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-17718:
--
Attachment: HBASE-17718.branch-1.001.patch
> Difference between RS's servername and its ephemeral node cause SSH stop
> working
>
>
> Key: HBASE-17718
> URL: https://issues.apache.org/jira/browse/HBASE-17718
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.2.4, 1.1.8
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17718.branch-1.001.patch,
> HBASE-17718.master.001.patch, HBASE-17718.master.002.patch,
> HBASE-17718.master.003.patch
>
>
> After HBASE-9593, RS put up an ephemeral node in ZK before reporting for
> duty. But if the hosts config (/etc/hosts) is different between master and
> RS, RS's serverName can be different from the one stored the ephemeral zk
> node. The email metioned in HBASE-13753
> (http://mail-archives.apache.org/mod_mbox/hbase-user/201505.mbox/%3CCANZDn9ueFEEuZMx=pZdmtLsdGLyZz=rrm1N6EQvLswYc1z-H=g...@mail.gmail.com%3E)
> is exactly what happened in our production env.
> But what the email didn't point out is that the difference between serverName
> in RS and zk node can cause SSH stop to work. as we can see from the code in
> {{RegionServerTracker}}
> {code}
> @Override
> public void nodeDeleted(String path) {
> if (path.startsWith(watcher.rsZNode)) {
> String serverName = ZKUtil.getNodeName(path);
> LOG.info("RegionServer ephemeral node deleted, processing expiration ["
> +
> serverName + "]");
> ServerName sn = ServerName.parseServerName(serverName);
> if (!serverManager.isServerOnline(sn)) {
> LOG.warn(serverName.toString() + " is not online or isn't known to
> the master."+
> "The latter could be caused by a DNS misconfiguration.");
> return;
> }
> remove(sn);
> this.serverManager.expireServer(sn);
> }
> }
> {code}
> The server will not be processed by SSH/ServerCrashProcedure. The regions on
> this server will not been assigned again until master restart or failover.
> I know HBASE-9593 was to fix the issue if RS report to duty and crashed
> before it can put up a zk node. It is a very rare case(And controllable， just
> fix the bug making rs to crash). But The issue I metioned can happened more
> often(and uncontrollable, can't be fixed in HBase, due to DNS, hosts config,
> etc.) and have more severe consequence.
> So here I offer some solutions to discuss:
> 1. Revert HBASE-9593 from all branches, Andrew Purtell has reverted it in
> branch-0.98
> 2. Abort RS if master return a different name, otherwise SSH can't work
> properly
> 3. Master accepts whatever servername reported by RS and don't change it.
> 4.correct the zk node if master return another name( idea from Ted Yu)
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17680) Run mini cluster through JNI in tests
[
https://issues.apache.org/jira/browse/HBASE-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898233#comment-15898233
]
Enis Soztutar commented on HBASE-17680:
---
Thanks Ted for the changes.
- Please remove ALL commented out code. This is the third time for the same
comment.
{code}
+ //hbase::Configuration conf;
+ // hbase::TestUtil *test_util = new hbase::TestUtil(2,
ClientTest::kDefHBaseConfPath.c_str());
{code}
- Refactor {{addColMid_}} -> {{add_col_mid_}}. and {{setConfMid_}}.
- These methods should take {{const std::string&}} instead of {{char *}}.
{code}
{{void WriteConf(jobject conf, const char *filepath);}}
+jbyteArray MiniCluster::StrToByteChar(const char *str) {
+TestUtil::TestUtil(int servers, const char *confPath)
{code}
- Remove:
{code}
+ LOG(INFO) << "retrieving " << key;
+ LOG(INFO) << "Got string " << val;
{code}
and maybe:
{code}
+ LOG(INFO) << "retrieved port " << port;
{code}
- Change these to check for NULL instead:
{code}
if (put == 0) {
+ if (mid == 0) {
+ if (n == 0) return NULL;
{code}
- I'll do the changes for client-test for not depending on the WriteConf() as a
follow up.
> Run mini cluster through JNI in tests
> -
>
> Key: HBASE-17680
> URL: https://issues.apache.org/jira/browse/HBASE-17680
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 17680.v14.txt, 17680.v17.txt, 17680.v18.txt,
> 17680.v1.txt, 17680.v20.txt, 17680.v22.txt, 17680.v23.txt, 17680.v26.txt,
> 17680.v27.txt, 17680.v28.txt, 17680.v29.txt, 17680.v30.txt, 17680.v31.txt,
> 17680.v3.txt, 17680.v8.txt
>
>
> Currently tests start local hbase cluster through hbase shell.
> There is less control over the configuration of the local cluster this way.
> This issue would replace hbase shell with JNI interface to mini cluster.
> We would have full control over the cluster behavior.
> Thanks to [~devaraj] who started this initiative.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17002) [Master] Report quotas and active violations on Master UI and JMX metrics
[ https://issues.apache.org/jira/browse/HBASE-17002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898173#comment-15898173 ] Ted Yu commented on HBASE-17002: lgtm > [Master] Report quotas and active violations on Master UI and JMX metrics > - > > Key: HBASE-17002 > URL: https://issues.apache.org/jira/browse/HBASE-17002 > Project: HBase > Issue Type: Sub-task > Components: master, UI >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17002.001.HBASE-16961.patch, > HBASE-17002.002.HBASE-16961.patch, HBASE-17002.003.HBASE-16961.patch, > TableQuotaDefined2.png, TableQuotaDefined.png > > > We should be able to view the Master UI and determine what quotas exist and > the quotas that are in violation to easily confirm/deny the state of > tables/namespaces. > The JMX metrics from the Master should also include the list of > tables/namespaces which have quotas whose violation policies are presently > being enforced. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17717) Incorrect ZK ACL set for HBase superuser
[
https://issues.apache.org/jira/browse/HBASE-17717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898170#comment-15898170
]
Josh Elser commented on HBASE-17717:
bq. +1 for branch-1.2
Thanks, Sean. I pushed this.
> Incorrect ZK ACL set for HBase superuser
>
>
> Key: HBASE-17717
> URL: https://issues.apache.org/jira/browse/HBASE-17717
> Project: HBase
> Issue Type: Bug
> Components: security, Zookeeper
>Reporter: Shreya Bhat
>Assignee: Josh Elser
> Fix For: 2.0.0, 1.4.0, 1.3.1, 1.1.10, 1.2.6
>
> Attachments: HBASE-17717.001.0.98.patch,
> HBASE-17717.001.branch-1.1.patch, HBASE-17717.001.patch
>
>
> Shreya was doing some testing of a deploy of HBase, verifying that the ZK
> ACLs were actually set as we expect (yay, security).
> She noticed that, in some cases, we were seeing multiple ACLs for the same
> user.
> {noformat}
> 'world,'anyone
> : r
> 'sasl,'hbase
> : cdrwa
> 'sasl,'hbase
> : cdrwa
> {noformat}
> After digging into this (and some insight from the mighty [~enis]), we
> realized that this was happening because of an overridden value for
> {{hbase.superuser}}. However, the ACL value doesn't match what we'd expect to
> see (as hbase.superuser was set to {{cstm-hbase}}).
> After digging into this code, it seems like the {{auth}} ACL scheme in
> ZooKeeper does not work as we expect.
> {code}
> if (superUser != null) {
> acls.add(new ACL(Perms.ALL, new Id("auth", superUser)));
> }
> {code}
> In the above, the {{"auth"}} scheme ignores any provided "subject" in the
> {{Id}} object. It *only* considers the authentication of the current
> connection. As such, our usage of this never actually sets the ACL for the
> superuser correctly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17568) Expire region reports in the HMaster
[ https://issues.apache.org/jira/browse/HBASE-17568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898165#comment-15898165 ] Josh Elser commented on HBASE-17568: Thanks, Ted! Will commit after precommit re-validates. > Expire region reports in the HMaster > > > Key: HBASE-17568 > URL: https://issues.apache.org/jira/browse/HBASE-17568 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17568.001.patch, > HBASE-17568.002.HBASE-16961.patch, HBASE-17568.003.HBASE-16961.patch > > > (From a TODO) > The RegionServers send reports of sizes from to the Master so the Master can > compute the "size" of each Table. > Presently, the Master has no way to expire these reports. Thus, reports for > tables that are deleted would be retained in memory. Retain the time at which > the report was received by the master, and then use that to determine when to > age it off. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17002) [Master] Report quotas and active violations on Master UI and JMX metrics
[ https://issues.apache.org/jira/browse/HBASE-17002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-17002: --- Attachment: HBASE-17002.003.HBASE-16961.patch .003 Addresses Ted's feedback that I had missed > [Master] Report quotas and active violations on Master UI and JMX metrics > - > > Key: HBASE-17002 > URL: https://issues.apache.org/jira/browse/HBASE-17002 > Project: HBase > Issue Type: Sub-task > Components: master, UI >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17002.001.HBASE-16961.patch, > HBASE-17002.002.HBASE-16961.patch, HBASE-17002.003.HBASE-16961.patch, > TableQuotaDefined2.png, TableQuotaDefined.png > > > We should be able to view the Master UI and determine what quotas exist and > the quotas that are in violation to easily confirm/deny the state of > tables/namespaces. > The JMX metrics from the Master should also include the list of > tables/namespaces which have quotas whose violation policies are presently > being enforced. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17680) Run mini cluster through JNI in tests
[ https://issues.apache.org/jira/browse/HBASE-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898154#comment-15898154 ] Ted Yu commented on HBASE-17680: "buck test --all" passed for patch v31. > Run mini cluster through JNI in tests > - > > Key: HBASE-17680 > URL: https://issues.apache.org/jira/browse/HBASE-17680 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 17680.v14.txt, 17680.v17.txt, 17680.v18.txt, > 17680.v1.txt, 17680.v20.txt, 17680.v22.txt, 17680.v23.txt, 17680.v26.txt, > 17680.v27.txt, 17680.v28.txt, 17680.v29.txt, 17680.v30.txt, 17680.v31.txt, > 17680.v3.txt, 17680.v8.txt > > > Currently tests start local hbase cluster through hbase shell. > There is less control over the configuration of the local cluster this way. > This issue would replace hbase shell with JNI interface to mini cluster. > We would have full control over the cluster behavior. > Thanks to [~devaraj] who started this initiative. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17002) [Master] Report quotas and active violations on Master UI and JMX metrics
[
https://issues.apache.org/jira/browse/HBASE-17002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898110#comment-15898110
]
Josh Elser commented on HBASE-17002:
Oops, sorry, Ted. I missed these comments from earlier. My apologies.
{quote}
NUM_TABLE_VIOLATING_QUOTAS_NAME seems better name.
Similar suggestion for NUM_VIOLATING_NS_QUOTAS_NAME : move NS ahead of
violating.
{quote}
I was trying to work around the phrasing here. It's not easy because a table
could be in violation from a namespace quota. Let me see if I can come up with
something better..
bq.
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterQuotaSourceFactoryImpl.java
: for classes in hbase-hadoop2-compat module, annotation for audience should
be added.
bq. Wrap lone lines.
Will do.
> [Master] Report quotas and active violations on Master UI and JMX metrics
> -
>
> Key: HBASE-17002
> URL: https://issues.apache.org/jira/browse/HBASE-17002
> Project: HBase
> Issue Type: Sub-task
> Components: master, UI
>Reporter: Josh Elser
>Assignee: Josh Elser
> Fix For: HBASE-16961
>
> Attachments: HBASE-17002.001.HBASE-16961.patch,
> HBASE-17002.002.HBASE-16961.patch, TableQuotaDefined2.png,
> TableQuotaDefined.png
>
>
> We should be able to view the Master UI and determine what quotas exist and
> the quotas that are in violation to easily confirm/deny the state of
> tables/namespaces.
> The JMX metrics from the Master should also include the list of
> tables/namespaces which have quotas whose violation policies are presently
> being enforced.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17602) Tweak chore delay/period defaults
[ https://issues.apache.org/jira/browse/HBASE-17602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-17602: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Tweak chore delay/period defaults > - > > Key: HBASE-17602 > URL: https://issues.apache.org/jira/browse/HBASE-17602 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Trivial > Fix For: HBASE-16961 > > Attachments: HBASE-17602.001.HBASE-16961.patch > > > In testing, found that the default chore periods/delays were a little too > lax. This resulted in quotas not being updated as soon as they really should > have been. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17718) Difference between RS's servername and its ephemeral node cause SSH stop working
[
https://issues.apache.org/jira/browse/HBASE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898089#comment-15898089
]
Hadoop QA commented on HBASE-17718:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 26s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 8s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 106m 52s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 152m 33s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856326/HBASE-17718.master.003.patch
|
| JIRA Issue | HBASE-17718 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 50a544df6f78 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / d2349c6 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5968/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5968/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Difference between RS's servername and its ephemeral node cause SSH stop
> working
>
>
> Key: HBASE-17718
> URL: https://issues.apache.org/jira/browse/HBASE-17718
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.2.4, 1.1.8
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments:
[jira] [Work stopped] (HBASE-17501) NullPointerException after Datanodes Decommissioned and Terminated
[ https://issues.apache.org/jira/browse/HBASE-17501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-17501 stopped by James Moore. --- > NullPointerException after Datanodes Decommissioned and Terminated > -- > > Key: HBASE-17501 > URL: https://issues.apache.org/jira/browse/HBASE-17501 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Patrick Dignan >Assignee: James Moore >Priority: Minor > Attachments: HBASE_17501.patch, HBASE_17501.patch, > HBASE_17501.patch.v2, HBASE_17501.patch.v3, HBASE_17501.patch.v4, > HBASE_17501.v3 > > > We recently encountered an interesting NullPointerException in HDFS that > bubbles up to HBase, and is resolved be restarting the regionserver. The > issue was exhibited while we were replacing a set of nodes in one of our > clusters with a new set. We did the following: > 1. Turn off the HBase balancer > 2. Gracefully move the regions off the nodes we’re shutting off using a tool > we wrote to do so > 3. Decommission the datanodes using the HDFS exclude hosts file and hdfs > dfsadmin -refreshNodes > 4. Wait for the datanodes to decommission fully > 5. Terminate the VMs the instances are running inside. > A few notes. We did not shutdown the datanode processes, and the nodes were > therefore not marked as dead by the namenode. We simply terminated the > datanode VM (in this case an AWS instance). The nodes were marked as > decommissioned. We are running our clusters with DNS, and when we terminate > VMs, the associated CName is removed and no longer resolves. The errors do > not seem to resolve without a restart. > After we did this, the remaining regionservers started throwing > NullPointerExceptions with the following stack trace: > 2017-01-19 23:09:05,638 DEBUG org.apache.hadoop.hbase.ipc.RpcServer: > RpcServer.RW.fifo.Q.read.handler=80,queue=14,port=60020: callId: 1727723891 > service: ClientService methodName: Scan size: 216 connection: > 172.16.36.128:31538 > java.io.IOException > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2214) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:204) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183) > Caused by: java.lang.NullPointerException > at org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1564) > at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:62) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1434) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1682) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1542) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:445) > at > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:266) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:642) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:592) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:294) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:199) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:343) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:198) > at > org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2106) > at > org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2096) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5544) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2569) > at > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2555) > at > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2536) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2405) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33738) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170) > ... 3 more -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17568) Expire region reports in the HMaster
[ https://issues.apache.org/jira/browse/HBASE-17568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898032#comment-15898032 ] Ted Yu commented on HBASE-17568: +1 > Expire region reports in the HMaster > > > Key: HBASE-17568 > URL: https://issues.apache.org/jira/browse/HBASE-17568 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17568.001.patch, > HBASE-17568.002.HBASE-16961.patch, HBASE-17568.003.HBASE-16961.patch > > > (From a TODO) > The RegionServers send reports of sizes from to the Master so the Master can > compute the "size" of each Table. > Presently, the Master has no way to expire these reports. Thus, reports for > tables that are deleted would be retained in memory. Retain the time at which > the report was received by the master, and then use that to determine when to > age it off. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17568) Expire region reports in the HMaster
[ https://issues.apache.org/jira/browse/HBASE-17568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-17568: --- Attachment: HBASE-17568.003.HBASE-16961.patch .003 Address Ted's review. > Expire region reports in the HMaster > > > Key: HBASE-17568 > URL: https://issues.apache.org/jira/browse/HBASE-17568 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17568.001.patch, > HBASE-17568.002.HBASE-16961.patch, HBASE-17568.003.HBASE-16961.patch > > > (From a TODO) > The RegionServers send reports of sizes from to the Master so the Master can > compute the "size" of each Table. > Presently, the Master has no way to expire these reports. Thus, reports for > tables that are deleted would be retained in memory. Retain the time at which > the report was received by the master, and then use that to determine when to > age it off. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17716) Formalize Scan Metric names
[ https://issues.apache.org/jira/browse/HBASE-17716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897984#comment-15897984 ] stack commented on HBASE-17716: --- To be clear, the METRIC annotation won't work. We want metric names to be public. Public annotations do not take audience qualifications since public means for everyone (makes sense I suppose). What we can do is make metric names public static finals. If in a private class, can annotate the metric name with audience public. For the few I looked at from your patch, they were from a public class (only checked the first few). So, to answer your question, I suggest you make public static final of metric names (with the _METRIC_NAME suffix) that you need and then we file a follow-up to do the rest. > Formalize Scan Metric names > --- > > Key: HBASE-17716 > URL: https://issues.apache.org/jira/browse/HBASE-17716 > Project: HBase > Issue Type: Bug > Components: metrics >Reporter: Karan Mehta >Assignee: Karan Mehta >Priority: Minor > Attachments: HBASE-17716.patch > > > HBase provides various metrics through the API's exposed by ScanMetrics > class. > The JIRA PHOENIX-3248 requires them to be surfaced through the Phoenix > Metrics API. Currently these metrics are referred via hard-coded strings, > which are not formal and can break the Phoenix API. Hence we need to refactor > the code to assign enums for these metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17707) New More Accurate Table Skew cost function/generator
[
https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897982#comment-15897982
]
Hadoop QA commented on HBASE-17707:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 48s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 107m 48s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 154m 5s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856319/HBASE-17707-07.patch |
| JIRA Issue | HBASE-17707 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux cf0903c85905 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / d2349c6 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5967/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5967/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> New More Accurate Table Skew cost function/generator
>
>
> Key: HBASE-17707
> URL: https://issues.apache.org/jira/browse/HBASE-17707
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 1.2.0
> Environment: CentOS Derivative with a derivative of the 3.18.43
> kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no
[jira] [Comment Edited] (HBASE-17716) Formalize Scan Metric names
[ https://issues.apache.org/jira/browse/HBASE-17716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897909#comment-15897909 ] Samarth Jain edited comment on HBASE-17716 at 3/6/17 7:47 PM: -- Question though - should we be addressing all the metrics that are exposed by HBase in this patch and see if they can be annotated with a @Metric annotation on a case by case basis? What about metrics in classes annotated as @Private ? Or should we just address Scan metrics in this patch and file a subsequent JIRA for rest of the metrics? was (Author: samarthjain): Question though - should we be addressing all the metrics that are exposed by HBase in this patch and see if they can be annotated with a @Metric annotation on a case by case basis? What about metrics in classes annotated as @Private ? > Formalize Scan Metric names > --- > > Key: HBASE-17716 > URL: https://issues.apache.org/jira/browse/HBASE-17716 > Project: HBase > Issue Type: Bug > Components: metrics >Reporter: Karan Mehta >Assignee: Karan Mehta >Priority: Minor > Attachments: HBASE-17716.patch > > > HBase provides various metrics through the API's exposed by ScanMetrics > class. > The JIRA PHOENIX-3248 requires them to be surfaced through the Phoenix > Metrics API. Currently these metrics are referred via hard-coded strings, > which are not formal and can break the Phoenix API. Hence we need to refactor > the code to assign enums for these metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17734) Guard against possibly copying the qualifier in the ScanDeleteTracker
[
https://issues.apache.org/jira/browse/HBASE-17734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897908#comment-15897908
]
Hadoop QA commented on HBASE-17734:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 45s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 93m 52s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 134m 13s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856317/HBASE-17734.v2.patch |
| JIRA Issue | HBASE-17734 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 2071445e3334 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / d2349c6 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5966/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5966/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Guard against possibly copying the qualifier in the ScanDeleteTracker
> -
>
> Key: HBASE-17734
> URL: https://issues.apache.org/jira/browse/HBASE-17734
> Project: HBase
> Issue Type: Improvement
>Reporter: CHIA-PING TSAI
>
[jira] [Commented] (HBASE-17716) Formalize Scan Metric names
[ https://issues.apache.org/jira/browse/HBASE-17716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897909#comment-15897909 ] Samarth Jain commented on HBASE-17716: -- Question though - should we be addressing all the metrics that are exposed by HBase in this patch and see if they can be annotated with a @Metric annotation on a case by case basis? What about metrics in classes annotated as @Private ? > Formalize Scan Metric names > --- > > Key: HBASE-17716 > URL: https://issues.apache.org/jira/browse/HBASE-17716 > Project: HBase > Issue Type: Bug > Components: metrics >Reporter: Karan Mehta >Assignee: Karan Mehta >Priority: Minor > Attachments: HBASE-17716.patch > > > HBase provides various metrics through the API's exposed by ScanMetrics > class. > The JIRA PHOENIX-3248 requires them to be surfaced through the Phoenix > Metrics API. Currently these metrics are referred via hard-coded strings, > which are not formal and can break the Phoenix API. Hence we need to refactor > the code to assign enums for these metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17716) Formalize Scan Metric names
[ https://issues.apache.org/jira/browse/HBASE-17716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897904#comment-15897904 ] Samarth Jain commented on HBASE-17716: -- +1 to the approach you have adopted, [~saint@gmail.com]. This is enough for us - metric names exposed as constants, with the _METRIC_NAME suffix as convention and having them annotated as @Metric to denote that they shouldn't be changed. > Formalize Scan Metric names > --- > > Key: HBASE-17716 > URL: https://issues.apache.org/jira/browse/HBASE-17716 > Project: HBase > Issue Type: Bug > Components: metrics >Reporter: Karan Mehta >Assignee: Karan Mehta >Priority: Minor > Attachments: HBASE-17716.patch > > > HBase provides various metrics through the API's exposed by ScanMetrics > class. > The JIRA PHOENIX-3248 requires them to be surfaced through the Phoenix > Metrics API. Currently these metrics are referred via hard-coded strings, > which are not formal and can break the Phoenix API. Hence we need to refactor > the code to assign enums for these metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17568) Expire region reports in the HMaster
[
https://issues.apache.org/jira/browse/HBASE-17568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897899#comment-15897899
]
Josh Elser commented on HBASE-17568:
bq. Name the class SizeSnapshotWithTimestamp ?
Sure. I was thinking {{Snapshot}} implied a notion of time, but that's very
well just me :)
bq. The method can be package private.
Good call. Will change.
> Expire region reports in the HMaster
>
>
> Key: HBASE-17568
> URL: https://issues.apache.org/jira/browse/HBASE-17568
> Project: HBase
> Issue Type: Sub-task
>Reporter: Josh Elser
>Assignee: Josh Elser
> Fix For: HBASE-16961
>
> Attachments: HBASE-17568.001.patch, HBASE-17568.002.HBASE-16961.patch
>
>
> (From a TODO)
> The RegionServers send reports of sizes from to the Master so the Master can
> compute the "size" of each Table.
> Presently, the Master has no way to expire these reports. Thus, reports for
> tables that are deleted would be retained in memory. Retain the time at which
> the report was received by the master, and then use that to determine when to
> age it off.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17602) Tweak chore delay/period defaults
[ https://issues.apache.org/jira/browse/HBASE-17602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897898#comment-15897898 ] Ted Yu commented on HBASE-17602: lgtm > Tweak chore delay/period defaults > - > > Key: HBASE-17602 > URL: https://issues.apache.org/jira/browse/HBASE-17602 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Trivial > Fix For: HBASE-16961 > > Attachments: HBASE-17602.001.HBASE-16961.patch > > > In testing, found that the default chore periods/delays were a little too > lax. This resulted in quotas not being updated as soon as they really should > have been. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17002) [Master] Report quotas and active violations on Master UI and JMX metrics
[ https://issues.apache.org/jira/browse/HBASE-17002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897896#comment-15897896 ] Ted Yu commented on HBASE-17002: Looking at 17002.002.HBASE-16961.patch , it seems my previous comments were not addressed. > [Master] Report quotas and active violations on Master UI and JMX metrics > - > > Key: HBASE-17002 > URL: https://issues.apache.org/jira/browse/HBASE-17002 > Project: HBase > Issue Type: Sub-task > Components: master, UI >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17002.001.HBASE-16961.patch, > HBASE-17002.002.HBASE-16961.patch, TableQuotaDefined2.png, > TableQuotaDefined.png > > > We should be able to view the Master UI and determine what quotas exist and > the quotas that are in violation to easily confirm/deny the state of > tables/namespaces. > The JMX metrics from the Master should also include the list of > tables/namespaces which have quotas whose violation policies are presently > being enforced. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
