[jira] [Commented] (HBASE-15789) PB related changes to work with offheap
[ https://issues.apache.org/jira/browse/HBASE-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15579208#comment-15579208 ] stack commented on HBASE-15789: --- There is an issue w/ the patch. Works the first time but the second run has an issue because won't apply atop a existing added file. My problem. Let me figure it. > PB related changes to work with offheap > --- > > Key: HBASE-15789 > URL: https://issues.apache.org/jira/browse/HBASE-15789 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15789.master.001.patch, HBASE-15789.patch, > HBASE-15789_V2.patch > > > This is an issue to brainstorm. Whether we go with pb 2.x or pb 3.0 and also > depends on the shading of protobuf classes. > We should also decide if we are going to fork the PB classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15789) PB related changes to work with offheap
[ https://issues.apache.org/jira/browse/HBASE-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15579096#comment-15579096 ] stack commented on HBASE-15789: --- Sorry about that [~chia7712] Thanks for the kick. I reverted for now. > PB related changes to work with offheap > --- > > Key: HBASE-15789 > URL: https://issues.apache.org/jira/browse/HBASE-15789 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15789.master.001.patch, HBASE-15789.patch, > HBASE-15789_V2.patch > > > This is an issue to brainstorm. Whether we go with pb 2.x or pb 3.0 and also > depends on the shading of protobuf classes. > We should also decide if we are going to fork the PB classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-15789) PB related changes to work with offheap
[ https://issues.apache.org/jira/browse/HBASE-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack reopened HBASE-15789: --- Reopen. Breaks build. > PB related changes to work with offheap > --- > > Key: HBASE-15789 > URL: https://issues.apache.org/jira/browse/HBASE-15789 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15789.master.001.patch, HBASE-15789.patch, > HBASE-15789_V2.patch > > > This is an issue to brainstorm. Whether we go with pb 2.x or pb 3.0 and also > depends on the shading of protobuf classes. > We should also decide if we are going to fork the PB classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578711#comment-15578711 ] Anastasia Braginsky commented on HBASE-16608: - FYI, we were able to run the PE tool with the recent patch successfully. Also we'll work on the one document explaining the CompactingMemStore for the HBASE-16849. > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-V01.patch, HBASE-16608-V03.patch, > HBASE-16608-V04.patch, HBASE-16608-V08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578701#comment-15578701 ] Anastasia Braginsky commented on HBASE-16608: - [~ram_krish], you are welcome to leave your comments on the RB and we will rethink the places of addition. > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-V01.patch, HBASE-16608-V03.patch, > HBASE-16608-V04.patch, HBASE-16608-V08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16815) Low scan ratio in RPC queue tuning triggers divide by zero exception
[
https://issues.apache.org/jira/browse/HBASE-16815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578669#comment-15578669
]
Lars George commented on HBASE-16815:
-
+1 on the patch, good idea to up the level since this is printed only once
anyways. Good on you [~zghaobac].
> Low scan ratio in RPC queue tuning triggers divide by zero exception
>
>
> Key: HBASE-16815
> URL: https://issues.apache.org/jira/browse/HBASE-16815
> Project: HBase
> Issue Type: Bug
> Components: regionserver, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Lars George
>Assignee: Guanghao Zhang
> Attachments: HBASE-16815.patch
>
>
> Trying the following settings:
> {noformat}
>
> hbase.ipc.server.callqueue.handler.factor
> 0.5
>
>
> hbase.ipc.server.callqueue.read.ratio
> 0.5
>
>
> hbase.ipc.server.callqueue.scan.ratio
> 0.1
>
> {noformat}
> With 30 default handlers, this means 15 queues. Further, it means 8 write
> queues and 7 read queues. 10% of that is {{0.7}} which is then floor'ed to
> {{0}}. The debug log confirms it, as the tertiary check omits the scan
> details when they are zero:
> {noformat}
> 2016-10-12 12:50:27,305 INFO [main] ipc.SimpleRpcScheduler: Using fifo as
> user call queue, count=15
> 2016-10-12 12:50:27,311 DEBUG [main] ipc.RWQueueRpcExecutor: FifoRWQ.default
> writeQueues=7 writeHandlers=15 readQueues=8 readHandlers=14
> {noformat}
> But the code in {{RWQueueRpcExecutor}} calls {{RpcExecutor.startHandler()}}
> nevertheless and that does this:
> {code}
> for (int i = 0; i < numHandlers; i++) {
> final int index = qindex + (i % qsize);
> String name = "RpcServer." + threadPrefix + ".handler=" +
> handlers.size() + ",queue=" +
> index + ",port=" + port;
> {code}
> The modulo triggers then
> {noformat}
> 2016-10-12 11:41:22,810 ERROR [main] master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:145)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:220)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:155)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:222)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:137)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2524)
> Caused by: java.lang.ArithmeticException: / by zero
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.startHandlers(RpcExecutor.java:125)
> at
> org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.startHandlers(RWQueueRpcExecutor.java:178)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.start(RpcExecutor.java:78)
> at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.start(SimpleRpcScheduler.java:272)
> at org.apache.hadoop.hbase.ipc.RpcServer.start(RpcServer.java:2212)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.start(RSRpcServices.java:1143)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:615)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:396)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.(HMasterCommandLine.java:312)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:140)
> ... 7 more
> {noformat}
> That causes the server to not even start. I would suggest we either skip the
> {{startHandler()}} call altogether, or make it zero aware.
> Another possible option is to reserve at least _one_ scan handler/queue when
> the scan ratio is greater than zero, but only of there is more than one read
> handler/queue to begin with. Otherwise the scan handler/queue should be zero
> and share the one read handler/queue.
> Makes sense?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15789) PB related changes to work with offheap
[
https://issues.apache.org/jira/browse/HBASE-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578650#comment-15578650
]
ChiaPing Tsai commented on HBASE-15789:
---
[~stack]
i get some compile error after committing this issue. The error are shown below.
{noformat}
[ERROR]
/root/hbase/hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/com/google/protobuf/ByteInputByteString.java:[125,34]
getOrCreateBuffer(int) has private access in
org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteBufferWriter
[ERROR]
/root/hbase/hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/com/google/protobuf/ByteInputByteString.java:[153,29]
incompatible types:
org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteInput cannot be
converted to byte[]
[ERROR]
/root/hbase/hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/com/google/protobuf/ByteInputByteString.java:[159,16]
no suitable method found for
partialIsValidUtf8(int,org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteInput,int,int)
method
org.apache.hadoop.hbase.shaded.com.google.protobuf.Utf8.partialIsValidUtf8(int,byte[],int,int)
is not applicable
(argument mismatch;
org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteInput cannot be
converted to byte[])
method
org.apache.hadoop.hbase.shaded.com.google.protobuf.Utf8.partialIsValidUtf8(int,java.nio.ByteBuffer,int,int)
is not applicable
(argument mismatch;
org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteInput cannot be
converted to java.nio.ByteBuffer)
[ERROR]
/root/hbase/hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/com/google/protobuf/ByteInputByteString.java:[247,41]
incompatible types:
org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteInput cannot be
converted to byte[]
{noformat}
Would you please take a look?
Thanks.
> PB related changes to work with offheap
> ---
>
> Key: HBASE-15789
> URL: https://issues.apache.org/jira/browse/HBASE-15789
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Reporter: ramkrishna.s.vasudevan
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15789.master.001.patch, HBASE-15789.patch,
> HBASE-15789_V2.patch
>
>
> This is an issue to brainstorm. Whether we go with pb 2.x or pb 3.0 and also
> depends on the shading of protobuf classes.
> We should also decide if we are going to fork the PB classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15789) PB related changes to work with offheap
[
https://issues.apache.org/jira/browse/HBASE-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578592#comment-15578592
]
Hadoop QA commented on HBASE-15789:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
1s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 10m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hbase-assembly {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 47s
{color} | {color:red} hbase-protocol-shaded in master has 22 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 26s
{color} | {color:red} hbase-protocol-shaded in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 25s
{color} | {color:red} hbase-protocol-shaded in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 25s {color}
| {color:red} hbase-protocol-shaded in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 10m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 8 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 3s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 44s
{color} | {color:red} The patch causes 24 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 23s
{color} | {color:red} The patch causes 24 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 2s
{color} | {color:red} The patch causes 24 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 40s
{color} | {color:red} The patch causes 24 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 19s
{color} | {color:red} The patch causes 24 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 59s
{color} | {color:red} The patch causes 24 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 37s
{color} | {color:red} The patch causes 24 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 16s
{color} | {color:red} The patch causes 24 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 55s
{color} | {color:red} The patch causes 24 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 14s
{color} | {color:red} hbase-protocol-shaded in the patch failed. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
[jira] [Commented] (HBASE-16724) Snapshot owner can't clone
[
https://issues.apache.org/jira/browse/HBASE-16724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578548#comment-15578548
]
Hudson commented on HBASE-16724:
FAILURE: Integrated in Jenkins build HBase-1.4 #472 (See
[https://builds.apache.org/job/HBase-1.4/472/])
HBASE-16724 Snapshot owner can't clone (ashishsinghi: rev
b7f283c6f6728238bb553c80aa6eafce0df0d650)
* (edit) src/main/asciidoc/_chapters/appendix_acl_matrix.adoc
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
> Snapshot owner can't clone
> --
>
> Key: HBASE-16724
> URL: https://issues.apache.org/jira/browse/HBASE-16724
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 2.0.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16724-V2.patch, HBASE-16724-V3.patch,
> HBASE-16724-branch-1.1.patch, HBASE-16724-branch-1.2.patch,
> HBASE-16724-branch-1.3.patch, HBASE-16724-branch-1.patch, HBASE-16724.patch
>
>
> Currently only Global admin has the access of cloning a snapshot.
> In AccessController,
> {code}
> @Override
> public void preCloneSnapshot(final
> ObserverContext ctx,
> final SnapshotDescription snapshot, final HTableDescriptor
> hTableDescriptor)
> throws IOException {
> requirePermission(getActiveUser(ctx), "cloneSnapshot " +
> snapshot.getName(), Action.ADMIN);
> }
> {code}
> Snapshot owner should be able to clone it, need to add a check like,
> {code}
> SnapshotDescriptionUtils.isSnapshotOwner(snapshot, user)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16851) User-facing documentation for the In-Memory Compaction feature
Edward Bortnikov created HBASE-16851: Summary: User-facing documentation for the In-Memory Compaction feature Key: HBASE-16851 URL: https://issues.apache.org/jira/browse/HBASE-16851 Project: HBase Issue Type: Sub-task Affects Versions: 2.0.0 Reporter: Edward Bortnikov -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14920) Compacting Memstore
[ https://issues.apache.org/jira/browse/HBASE-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578538#comment-15578538 ] Edward Bortnikov commented on HBASE-14920: -- Opened the doc for commenting, thanks for suggesting [~stack]. > Compacting Memstore > --- > > Key: HBASE-14920 > URL: https://issues.apache.org/jira/browse/HBASE-14920 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Fix For: 2.0.0 > > Attachments: HBASE-14920-V01.patch, HBASE-14920-V02.patch, > HBASE-14920-V03.patch, HBASE-14920-V04.patch, HBASE-14920-V05.patch, > HBASE-14920-V06.patch, HBASE-14920-V07.patch, HBASE-14920-V08.patch, > HBASE-14920-V09.patch, HBASE-14920-V10.patch, move.to.junit4.patch > > > Implementation of a new compacting memstore with non-optimized immutable > segment representation -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16274) Add more peer tests to replication_admin_test
[
https://issues.apache.org/jira/browse/HBASE-16274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578466#comment-15578466
]
Hudson commented on HBASE-16274:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1792 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1792/])
HBASE-16274 Add more peer tests to replication_admin_test (Guanghao (tedyu: rev
76e7c05474fbd59fe98e499c4f69e07923473e7c)
* (edit) hbase-shell/src/test/ruby/hbase/replication_admin_test.rb
> Add more peer tests to replication_admin_test
> -
>
> Key: HBASE-16274
> URL: https://issues.apache.org/jira/browse/HBASE-16274
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Guanghao Zhang
>Priority: Minor
> Labels: replication
> Attachments: HBASE-16274-v1.patch, HBASE-16274.patch
>
>
> In master build #1275,
> https://builds.apache.org/job/HBase-TRUNK_matrix/1275/jdk=latest1.8,label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.client/TestReplicationShell/testRunShellTests/:
> {code}
> 1) Failure:
> test_add_peer:_multiple_zk_cluster_key_and_table_cfs_-_peer_config(Hbase::ReplicationAdminTest)
> [./src/test/ruby/hbase/replication_admin_test.rb:143:in
> `test_add_peer:_multiple_zk_cluster_key_and_table_cfs_-_peer_config'
> org/jruby/RubyProc.java:270:in `call'
> org/jruby/RubyKernel.java:2105:in `send'
> org/jruby/RubyArray.java:1620:in `each'
> org/jruby/RubyArray.java:1620:in `each']:
> <"default.table1;default.table3:cf1,cf2;default.table2:cf1"> expected but was
> <"default.table3:cf1,cf2;default.table2:cf1;default.table1">.
> {code}
> As far as I can see, the two peer table CFs are the same, except for ordering.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15789) PB related changes to work with offheap
[ https://issues.apache.org/jira/browse/HBASE-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578441#comment-15578441 ] stack commented on HBASE-15789: --- Nvm. One commit only. My commit included adding this patching mechanism, the patch in src/main/patches, and then the result of the running of the -Pgenerate-shaded-classes step all in the one go. > PB related changes to work with offheap > --- > > Key: HBASE-15789 > URL: https://issues.apache.org/jira/browse/HBASE-15789 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15789.master.001.patch, HBASE-15789.patch, > HBASE-15789_V2.patch > > > This is an issue to brainstorm. Whether we go with pb 2.x or pb 3.0 and also > depends on the shading of protobuf classes. > We should also decide if we are going to fork the PB classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15789) PB related changes to work with offheap
[ https://issues.apache.org/jira/browse/HBASE-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15789: -- Resolution: Fixed Hadoop Flags: Reviewed Release Note: Adds means of patching our checked-in protobuf as last part of our offline -Pgenerate-shaded-classes step. Patches found in the new src/main/patches directory are all applied as the last step when you run a build with the -Pgenerate-shaded-classes profile. This commit also includes an actual patch that adds ByteInput to mimic pb3.1's ByteOutput (src/main/patches/HBASE-15789_V2.patch attached here). (was: Adds means of patching our checked-in protobuf as last part of our offline -Pgenerate-shaded-classes step. As part of introduction of the above, it exercises the facility with a protobuf patch to add ByteInput support.) Status: Resolved (was: Patch Available) Pushed to master. Required two commits. Once to add this facility and the patch to pb and then a second commit of changes that are the result of the -Pgenerate-shaded-classes step. > PB related changes to work with offheap > --- > > Key: HBASE-15789 > URL: https://issues.apache.org/jira/browse/HBASE-15789 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15789.master.001.patch, HBASE-15789.patch, > HBASE-15789_V2.patch > > > This is an issue to brainstorm. Whether we go with pb 2.x or pb 3.0 and also > depends on the shading of protobuf classes. > We should also decide if we are going to fork the PB classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15789) PB related changes to work with offheap
[ https://issues.apache.org/jira/browse/HBASE-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578421#comment-15578421 ] stack commented on HBASE-15789: --- bq. Even the new classes in the patch can be inside src/patches/xxx.patch file? Yes. bq. So when dev download src code and setup in IDE, this step will happen so that they can see these patches classes right? No. The patch is applied as part of the -Pgenerate-shaded-classes step done when you change protobuf or if you change these patches or if you change the protobuf version. Let me commit this. Means checking in this new facility, your patch as is, and then after running the -Pgenerate-shaded-classes step, all the changes (patched protobuf lib in this case). > PB related changes to work with offheap > --- > > Key: HBASE-15789 > URL: https://issues.apache.org/jira/browse/HBASE-15789 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15789.master.001.patch, HBASE-15789.patch, > HBASE-15789_V2.patch > > > This is an issue to brainstorm. Whether we go with pb 2.x or pb 3.0 and also > depends on the shading of protobuf classes. > We should also decide if we are going to fork the PB classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16274) Add more peer tests to replication_admin_test
[
https://issues.apache.org/jira/browse/HBASE-16274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578198#comment-15578198
]
Ted Yu commented on HBASE-16274:
Mind attaching patch for branch-1 ?
Thanks
> Add more peer tests to replication_admin_test
> -
>
> Key: HBASE-16274
> URL: https://issues.apache.org/jira/browse/HBASE-16274
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Guanghao Zhang
>Priority: Minor
> Labels: replication
> Attachments: HBASE-16274-v1.patch, HBASE-16274.patch
>
>
> In master build #1275,
> https://builds.apache.org/job/HBase-TRUNK_matrix/1275/jdk=latest1.8,label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.client/TestReplicationShell/testRunShellTests/:
> {code}
> 1) Failure:
> test_add_peer:_multiple_zk_cluster_key_and_table_cfs_-_peer_config(Hbase::ReplicationAdminTest)
> [./src/test/ruby/hbase/replication_admin_test.rb:143:in
> `test_add_peer:_multiple_zk_cluster_key_and_table_cfs_-_peer_config'
> org/jruby/RubyProc.java:270:in `call'
> org/jruby/RubyKernel.java:2105:in `send'
> org/jruby/RubyArray.java:1620:in `each'
> org/jruby/RubyArray.java:1620:in `each']:
> <"default.table1;default.table3:cf1,cf2;default.table2:cf1"> expected but was
> <"default.table3:cf1,cf2;default.table2:cf1;default.table1">.
> {code}
> As far as I can see, the two peer table CFs are the same, except for ordering.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[
https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578087#comment-15578087
]
Duo Zhang commented on HBASE-16835:
---
{code:title=ClusterRegistry.java}
interface ClusterRegistry extends Closeable {
RegionLocations getMetaRegionLocation();
/**
* Should only be called once.
*
* The upper layer should store this value somewhere as it will not be change
any more.
*/
String getClusterId();
int getCurrentNrHRS();
ServerName getMasterAddress();
int getMasterInfoPort();
@Override
void close();
}
{code}
This is the original ClusterRegistry interface introduced in HBASE-15921. I
added all the zk related operations needed at client side to it.
The getClusterId will only be called once when start up so it can be blocking
and read directly from zk. And for getCurrentNrHRS, I think it is not very
important so let's discuss it later. The getMasterInfoPort gets the same thing
from zookeeper with getMasterAddress. So actually, the most important methods
are getMetaRegionLocation and getMasterAddress.
For meta location, we will also cache it meta cache. And for master address, we
have a logic to decide when to fetch the new address from zk. And usually, a
client will cache all the region locations needed so it even does not need to
know the meta location change as it will not access the meta table for a long
time. And for master, it is also rarely touched from client. So I think it is
reasonable to not rely on watcher to update the value as it may cause
unnecessary network traffic.
But I think the API of meta cache and master address tracker should be changed.
We have two timeouts right now, one for the normal operations, and the other
one for the meta operations. I think we should have a larger timeout for meta
operations because if we done then we could cache it. But in the old blocking
world, it is a not a good idea to have a larger timeout in a sub stage... In
the asynchronous world, we could return a CompletableFuture and use
HashedWheelTimer to trigger timeout. So it is possible to have a larger timeout
in a sub stage as upper layer has the ability to timeout before the actual
process done.
> Revisit the zookeeper usage at client side
> --
>
> Key: HBASE-16835
> URL: https://issues.apache.org/jira/browse/HBASE-16835
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>
> Watcher or not.
> Curator or not.
> Keep connection or not.
> ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16834) Implement AsyncConnectionFactory
[ https://issues.apache.org/jira/browse/HBASE-16834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16834: -- Release Note: Introduce AsyncConnectionFactory for instantiating AsyncConnection. The default implementation is org.apache.hadoop.hbase.client.AsyncConnectionImpl. You can use 'hbase.client.async.connection.impl' to plug in your own AsyncConnection implementation. > Implement AsyncConnectionFactory > > > Key: HBASE-16834 > URL: https://issues.apache.org/jira/browse/HBASE-16834 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16834.patch > > > Or AsyncConnectionManager? > The name depends on whether we want to implement the connection management > logic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16834) Implement AsyncConnectionFactory
[ https://issues.apache.org/jira/browse/HBASE-16834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16834: -- Affects Version/s: 2.0.0 Fix Version/s: 2.0.0 Component/s: Client > Implement AsyncConnectionFactory > > > Key: HBASE-16834 > URL: https://issues.apache.org/jira/browse/HBASE-16834 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16834.patch > > > Or AsyncConnectionManager? > The name depends on whether we want to implement the connection management > logic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16834) Implement AsyncConnectionFactory
[ https://issues.apache.org/jira/browse/HBASE-16834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16834: -- Attachment: HBASE-16834.patch > Implement AsyncConnectionFactory > > > Key: HBASE-16834 > URL: https://issues.apache.org/jira/browse/HBASE-16834 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang > Attachments: HBASE-16834.patch > > > Or AsyncConnectionManager? > The name depends on whether we want to implement the connection management > logic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-16834) Implement AsyncConnectionFactory
[ https://issues.apache.org/jira/browse/HBASE-16834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-16834: - Assignee: Duo Zhang > Implement AsyncConnectionFactory > > > Key: HBASE-16834 > URL: https://issues.apache.org/jira/browse/HBASE-16834 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang > Attachments: HBASE-16834.patch > > > Or AsyncConnectionManager? > The name depends on whether we want to implement the connection management > logic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16724) Snapshot owner can't clone
[
https://issues.apache.org/jira/browse/HBASE-16724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Singhi updated HBASE-16724:
--
Fix Version/s: 1.4.0
Pushed to branch-1.
On branch-1.3 the modified test in the patch failed. Didn't check further
branches.
{code}
---
T E S T S
---
Running org.apache.hadoop.hbase.security.access.TestAccessController
Tests run: 61, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 88.736 sec <<<
FAILURE! - in org.apache.hadoop.hbase.security.access.TestAccessController
testSnapshot(org.apache.hadoop.hbase.security.access.TestAccessController)
Time elapsed: 0.035 sec <<< ERROR!
java.lang.NullPointerException: null
at
org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils.isSnapshotOwner(SnapshotDescriptionUtils.java:364)
at
org.apache.hadoop.hbase.security.access.AccessController.preCloneSnapshot(AccessController.java:1335)
at
org.apache.hadoop.hbase.security.access.TestAccessController$70.run(TestAccessController.java:2053)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at
org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:340)
at
org.apache.hadoop.hbase.security.access.SecureTestUtil.verifyAllowed(SecureTestUtil.java:177)
at
org.apache.hadoop.hbase.security.access.SecureTestUtil.verifyAllowed(SecureTestUtil.java:193)
at
org.apache.hadoop.hbase.security.access.TestAccessController.testSnapshot(TestAccessController.java:2063)
Results :
Tests in error:
TestAccessController.testSnapshot:2063->SecureTestUtil.verifyAllowed:193->SecureTestUtil.verifyAllowed:177
▒ NullPointer
Tests run: 61, Failures: 0, Errors: 1, Skipped: 0
{code}
Next time please run test locally also before attaching the patch.
> Snapshot owner can't clone
> --
>
> Key: HBASE-16724
> URL: https://issues.apache.org/jira/browse/HBASE-16724
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 2.0.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16724-V2.patch, HBASE-16724-V3.patch,
> HBASE-16724-branch-1.1.patch, HBASE-16724-branch-1.2.patch,
> HBASE-16724-branch-1.3.patch, HBASE-16724-branch-1.patch, HBASE-16724.patch
>
>
> Currently only Global admin has the access of cloning a snapshot.
> In AccessController,
> {code}
> @Override
> public void preCloneSnapshot(final
> ObserverContext ctx,
> final SnapshotDescription snapshot, final HTableDescriptor
> hTableDescriptor)
> throws IOException {
> requirePermission(getActiveUser(ctx), "cloneSnapshot " +
> snapshot.getName(), Action.ADMIN);
> }
> {code}
> Snapshot owner should be able to clone it, need to add a check like,
> {code}
> SnapshotDescriptionUtils.isSnapshotOwner(snapshot, user)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
