[jira] [Created] (HBASE-19340) SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell
zhaoyuan created HBASE-19340: Summary: SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell Key: HBASE-19340 URL: https://issues.apache.org/jira/browse/HBASE-19340 Project: HBase Issue Type: Bug Affects Versions: 1.2.6 Reporter: zhaoyuan Fix For: 1.2.8 Recently I wanna try to alter the split policy for a table on my cluster which version is 1.2.6 and as far as I know The SPLIT_POLICY is an attribute of the HTable so I run the command in hbase shell console below. alter 'tablex',SPLIT_POLICY => 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' However, It gave the information like this and I confused Unknown argument ignored: SPLIT_POLICY Updating all regions with the new schema... So I check the source code That admin.rb might miss the setting for this argument . htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if arg[MAX_FILESIZE] htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] ... So I think it may be a bug in hbase-server ,is it? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265019#comment-16265019
]
Yu Li commented on HBASE-19338:
---
bq. I'm a latecomer to this party
Please check HBASE-14460 if you are interested about the history (smile)
[~chia7712]
Comparing performance between latest branch-1 and branch-2 is another task to
follow, let's keep the discussion on current issue first, and file other JIRAs
for the new mission. :-)
So, anything blocking us to get this one in? Seems we need to re-attach the
latest patch and get the HadoopQA report? (per Ted mentioned, last run failed
to generate a report?) Thanks.
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, HBASE-19338.master.001.patch,
> HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265011#comment-16265011
]
binlijin commented on HBASE-19338:
--
V3 is ok with me.
YCSB :
1 regionserver.
4 client machine, every machine run 10 ycsb process and every yscb use 100
threads.
Regionserver config is :
"-Xmx36864m -Xms36864m -Xmn10240m -XX:SurvivorRatio=3
-XX:MaxTenuringThreshold=15 -Xss256k -XX:+UseParNewGC
-XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=63
-XX:ParallelGCThreads=16"
dfs.client.block.write.replace-datanode-on-failure.enable
true
dfs.client.read.shortcircuit
true
dfs.client.socket-timeout
3
dfs.datanode.socket.write.timeout
3
dfs.domain.socket.path
/var/run/hadoop-hdfs/dn._PORT
hbase.balancer.period
360
hbase.block.data.cachecompressed
false
hbase.bucketcache.combinedcache.enabled
true
hbase.bucketcache.ioengine
offheap
hbase.bucketcache.size
12288
hbase.bucketcache.writer.queuelength
64
hbase.bucketcache.writer.threads
3
hbase.bulk.assignment.threadpool.size
100
hbase.bulkload.wait.duration
1
hbase.client.keyvalue.maxsize
41943040
hbase.client.log.batcherrors.details
true
hbase.client.operation.timeout
18
hbase.client.pause
1000
hbase.client.retries.number
34
hbase.client.rpc.codec
org.apache.hadoop.hbase.codec.KeyValueCodecWithTags
hbase.client.scanner.caching
8
hbase.client.scanner.max.result.size
2097152
hbase.client.scanner.timeout.period
18
hbase.client.write.autoflush
true
hbase.client.write.buffer
2097152
hbase.cluster.distributed
true
hbase.coprocessor.abortonerror
false
hbase.defaults.for.version.skip
true
hbase.dfs.client.read.shortcircuit.buffer.size
16384
hbase.hlog.split.skip.errors
false
hbase.hregion.majorcompaction
60480
hbase.hregion.max.filesize
8589934592
hbase.hregion.memstore.block.multiplier
2
hbase.hregion.memstore.chunkpool.initialsize
1
hbase.hregion.memstore.flush.size
268435456
hbase.hregion.memstore.mslab.enabled
true
hbase.hstore.block.storage.policy
ALL_SSD
hbase.hstore.blockingStoreFiles
20
hbase.hstore.blockingWaitTime
9
hbase.hstore.compaction.kv.max
128
hbase.hstore.compaction.max
10
hbase.hstore.compaction.min
4
hbase.hstore.compaction.throughput.higher.bound
167772160
hbase.hstore.compaction.throughput.lower.bound
167772160
hbase.hstore.compaction.throughput.offpeak
167772160
hbase.hstore.compactionThreshold
3
hbase.hstore.flush.retries.number
120
hbase.hstore.flusher.count
2
hbase.hstore.open.and.close.threads.max
10
hbase.ipc.client.connection.maxidletime
6
hbase.ipc.server.callqueue.handler.factor
0.1
hbase.ipc.server.callqueue.read.share
0
hbase.lease.recovery.dfs.timeout
23000
hbase.local.dir
${hbase.tmp.dir}/local
hbase.master.assignment.timeoutmonitor.timeout
90
hbase.master.distributed.log.replay
false
hbase.master.executor.serverops.threads
30
hbase.master.info.bindAddress
0.0.0.0
hbase.master.info.port
60010
hbase.master.loadbalancer.class
org.apache.hadoop.hbase.master.balancer.SimpleLoadBalancer
hbase.master.port
60100
hbase.master.ui.readonly
true
hbase.offheapcache.minblocksize
16384
hbase.regions.slop
0.1
hbase.regionserver.compaction.private.readers
true
hbase.regionserver.executor.closeregion.threads
10
hbase.regionserver.executor.openregion.threads
10
hbase.regionserver.global.memstore.lowerLimit
0.35
hbase.regionserver.global.memstore.size
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265009#comment-16265009
]
Chia-Ping Tsai commented on HBASE-19338:
{quote}
In fact, 1.x is much slower than 0.98 even after several minor and patch
releases. We hope this won't happen again for 2.0
{quote}
I'm a latecomer to this party. :(
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, HBASE-19338.master.001.patch,
> HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[ https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265007#comment-16265007 ] Chia-Ping Tsai commented on HBASE-19092: bq. I think we can add a release note after HBASE-19112? Or do we need to add one here? The story happening here is complicated, so a brief note is necessary for the readers I think. > Make Tag IA.LimitedPrivate and expose for CPs > - > > Key: HBASE-19092 > URL: https://issues.apache.org/jira/browse/HBASE-19092 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19092-branch-2.patch, > HBASE-19092-branch-2_5.patch, HBASE-19092-branch-2_5.patch, > HBASE-19092.branch-2.0.02.patch, HBASE-19092_001-branch-2.patch, > HBASE-19092_001.patch, HBASE-19092_002-branch-2.patch, HBASE-19092_002.patch, > HBASE-19092_004.patch, HBASE-19092_005.patch, HBASE-19092_005_branch_2.patch, > HBASE-19092_3.patch, HBASE-19092_4.patch > > > We need to make tags as LimitedPrivate as some use cases are trying to use > tags like timeline server. The same topic was discussed in dev@ and also in > HBASE-18995. > Shall we target this for beta1 - cc [~saint@gmail.com]. > So once we do this all related Util methods and APIs should also move to > LimitedPrivate Util classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19336) Improve rsgroup to allow assign all tables within a specified namespace by only writing namespace
[
https://issues.apache.org/jira/browse/HBASE-19336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xinxin fan updated HBASE-19336:
---
Attachment: HBASE-19336-master-V3.patch
> Improve rsgroup to allow assign all tables within a specified namespace by
> only writing namespace
> -
>
> Key: HBASE-19336
> URL: https://issues.apache.org/jira/browse/HBASE-19336
> Project: HBase
> Issue Type: Improvement
> Components: rsgroup
>Affects Versions: 2.0.0-alpha-4
>Reporter: xinxin fan
>Assignee: xinxin fan
> Attachments: HBASE-19336-master-V2.patch,
> HBASE-19336-master-V3.patch, HBASE-19336-master.patch
>
>
> Currently, use can only assign tables within a namespace from one group to
> another by writing all table names in move_tables_rsgroup command. Allowing
> to assign all tables within a specifed namespace by only wirting namespace
> name is useful.
> Usage as follows:
> {code:java}
> hbase(main):055:0> move_namespaces_rsgroup 'dest_rsgroup',['ns1']
> Took 2.2211 seconds
> {code}
> {code:java}
> hbase(main):051:0* move_servers_namespaces_rsgroup
> 'dest_rsgroup',['hbase39.lt.163.org:60020'],['ns1','ns2']
> Took 15.3710 seconds
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265004#comment-16265004
]
Duo Zhang commented on HBASE-19338:
---
{quote}
Our master branch is slower than branch-1
{quote}
This is the normal case :)
In fact, 1.x is much slower than 0.98 even after several minor and patch
releases. We hope this won't happen again for 2.0, but, performance is always
the last thing to be considered for a release as we need to confirm the
correctness first, especially for a storage system. Do not loss data is first
priority.
The good news is that, the guys from Alibaba also target 2.0 as their next
major hbase version, and they are really good at tuning the performance. So I
believe that this time we can get a good performance at the final 2.0.0 release.
Thanks.
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, HBASE-19338.master.001.patch,
> HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[ https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264999#comment-16264999 ] ramkrishna.s.vasudevan commented on HBASE-19092: I think we can add a release note after HBASE-19112? Or do we need to add one here? > Make Tag IA.LimitedPrivate and expose for CPs > - > > Key: HBASE-19092 > URL: https://issues.apache.org/jira/browse/HBASE-19092 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19092-branch-2.patch, > HBASE-19092-branch-2_5.patch, HBASE-19092-branch-2_5.patch, > HBASE-19092.branch-2.0.02.patch, HBASE-19092_001-branch-2.patch, > HBASE-19092_001.patch, HBASE-19092_002-branch-2.patch, HBASE-19092_002.patch, > HBASE-19092_004.patch, HBASE-19092_005.patch, HBASE-19092_005_branch_2.patch, > HBASE-19092_3.patch, HBASE-19092_4.patch > > > We need to make tags as LimitedPrivate as some use cases are trying to use > tags like timeline server. The same topic was discussed in dev@ and also in > HBASE-18995. > Shall we target this for beta1 - cc [~saint@gmail.com]. > So once we do this all related Util methods and APIs should also move to > LimitedPrivate Util classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[ https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-19092: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Make Tag IA.LimitedPrivate and expose for CPs > - > > Key: HBASE-19092 > URL: https://issues.apache.org/jira/browse/HBASE-19092 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19092-branch-2.patch, > HBASE-19092-branch-2_5.patch, HBASE-19092-branch-2_5.patch, > HBASE-19092.branch-2.0.02.patch, HBASE-19092_001-branch-2.patch, > HBASE-19092_001.patch, HBASE-19092_002-branch-2.patch, HBASE-19092_002.patch, > HBASE-19092_004.patch, HBASE-19092_005.patch, HBASE-19092_005_branch_2.patch, > HBASE-19092_3.patch, HBASE-19092_4.patch > > > We need to make tags as LimitedPrivate as some use cases are trying to use > tags like timeline server. The same topic was discussed in dev@ and also in > HBASE-18995. > Shall we target this for beta1 - cc [~saint@gmail.com]. > So once we do this all related Util methods and APIs should also move to > LimitedPrivate Util classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[ https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-19092: --- Attachment: HBASE-19092_005_branch_2.patch HBASE-19092_005.patch Patch that I committed. Thanks all for the reviews and feedback. Will start working on RawCell as part of HBASE-19112. > Make Tag IA.LimitedPrivate and expose for CPs > - > > Key: HBASE-19092 > URL: https://issues.apache.org/jira/browse/HBASE-19092 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19092-branch-2.patch, > HBASE-19092-branch-2_5.patch, HBASE-19092-branch-2_5.patch, > HBASE-19092.branch-2.0.02.patch, HBASE-19092_001-branch-2.patch, > HBASE-19092_001.patch, HBASE-19092_002-branch-2.patch, HBASE-19092_002.patch, > HBASE-19092_004.patch, HBASE-19092_005.patch, HBASE-19092_005_branch_2.patch, > HBASE-19092_3.patch, HBASE-19092_4.patch > > > We need to make tags as LimitedPrivate as some use cases are trying to use > tags like timeline server. The same topic was discussed in dev@ and also in > HBASE-18995. > Shall we target this for beta1 - cc [~saint@gmail.com]. > So once we do this all related Util methods and APIs should also move to > LimitedPrivate Util classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19334) User.runAsLoginUser not work in AccessController because it use a short circuited connection
[
https://issues.apache.org/jira/browse/HBASE-19334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264996#comment-16264996
]
Guanghao Zhang commented on HBASE-19334:
bq. when short circuit connection is been used, the user is changing depending
on target server or which API is been used.
If the target server is the local RS, we still can't change the user to super
user which started RS process? But if call User.runAsLoginUser, I thought we
should change the user, but need take a try for this. I don't know if it is
possible...
> User.runAsLoginUser not work in AccessController because it use a short
> circuited connection
>
>
> Key: HBASE-19334
> URL: https://issues.apache.org/jira/browse/HBASE-19334
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-19334.master.001.patch
>
>
> The short-circuited connection will bypass the RPC and the RPC context didn't
> change. So it still use the old RPC user to write ACL table and
> User.runAsLoginUser not work.
> AccessController's grant method.
> {code}
> User.runAsLoginUser(new PrivilegedExceptionAction() {
> @Override
> public Void run() throws Exception {
> // regionEnv is set at #start. Hopefully not null at this point.
> try (Table table = regionEnv.getConnection().
> getTable(AccessControlLists.ACL_TABLE_NAME)) {
>
> AccessControlLists.addUserPermission(regionEnv.getConfiguration(), perm,
> table,
> request.getMergeExistingPermissions());
> }
> return null;
> }
> });
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264992#comment-16264992
]
Chia-Ping Tsai commented on HBASE-19338:
{quote}
if i use multiwal and 1 wal and with the patch the qps is 11.2W.
I also test the latest branch-1 with commit
20d811121fb38ea2fc3871dcf4b03593bd4d6b7e, we can get 12W qps.
{quote}
Our master branch is slower than branch-1...Do you mind sharing the ycsb and
env config? Perhaps there are other performance issues.
is V3 ok to you? [~aoxiang]
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, HBASE-19338.master.001.patch,
> HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18127) Enable state to be passed between the region observer coprocessor hook calls
[ https://issues.apache.org/jira/browse/HBASE-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264984#comment-16264984 ] Anoop Sam John edited comment on HBASE-18127 at 11/24/17 6:49 AM: -- May be there are no such cases now? But it is possible to come. Ya what I was interested in is a general solution. We have this ThreadLocal issue some other places also. That, part of code executed by a thread from a pool. We need a generic solution. If u can try out some thing general which we can use throughout, that would be awesome. Good one Abhishek was (Author: anoop.hbase): May be there are no such cases now? But it is possible to come. Ya what I was interested in is a general solution. We have this ThreadLocal issue some other places also. That part of code executed by a thread a pool. We need a generic solution. If u can try out some thing general which we can use throughout, that would be awesome. Good one Abhishek > Enable state to be passed between the region observer coprocessor hook calls > > > Key: HBASE-18127 > URL: https://issues.apache.org/jira/browse/HBASE-18127 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-18127.master.001.patch, > HBASE-18127.master.002.patch, HBASE-18127.master.002.patch, > HBASE-18127.master.003.patch, HBASE-18127.master.004.patch, > HBASE-18127.master.005.patch, HBASE-18127.master.005.patch, > HBASE-18127.master.006.patch > > > Allow regionobserver to optionally skip postPut/postDelete when > postBatchMutate was called. > Right now a RegionObserver can only statically implement one or the other. In > scenarios where we need to work sometimes on the single postPut and > postDelete hooks and sometimes on the batchMutate hooks, there is currently > no place to convey this information to the single hooks. I.e. the work has > been done in the batch, skip the single hooks. > There are various solutions: > 1. Allow some state to be passed _per operation_. > 2. Remove the single hooks and always only call batch hooks (with a default > wrapper for the single hooks). > 3. more? > [~apurtell], what we had discussed a few days back. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18127) Enable state to be passed between the region observer coprocessor hook calls
[ https://issues.apache.org/jira/browse/HBASE-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264984#comment-16264984 ] Anoop Sam John commented on HBASE-18127: May be there are no such cases now? But it is possible to come. Ya what I was interested in is a general solution. We have this ThreadLocal issue some other places also. That part of code executed by a thread a pool. We need a generic solution. If u can try out some thing general which we can use throughout, that would be awesome. Good one Abhishek > Enable state to be passed between the region observer coprocessor hook calls > > > Key: HBASE-18127 > URL: https://issues.apache.org/jira/browse/HBASE-18127 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-18127.master.001.patch, > HBASE-18127.master.002.patch, HBASE-18127.master.002.patch, > HBASE-18127.master.003.patch, HBASE-18127.master.004.patch, > HBASE-18127.master.005.patch, HBASE-18127.master.005.patch, > HBASE-18127.master.006.patch > > > Allow regionobserver to optionally skip postPut/postDelete when > postBatchMutate was called. > Right now a RegionObserver can only statically implement one or the other. In > scenarios where we need to work sometimes on the single postPut and > postDelete hooks and sometimes on the batchMutate hooks, there is currently > no place to convey this information to the single hooks. I.e. the work has > been done in the batch, skip the single hooks. > There are various solutions: > 1. Allow some state to be passed _per operation_. > 2. Remove the single hooks and always only call batch hooks (with a default > wrapper for the single hooks). > 3. more? > [~apurtell], what we had discussed a few days back. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264982#comment-16264982
]
binlijin commented on HBASE-19338:
--
I also test the latest branch-1 with commit
20d811121fb38ea2fc3871dcf4b03593bd4d6b7e, we can get 12W qps.
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, HBASE-19338.master.001.patch,
> HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264983#comment-16264983 ] ramkrishna.s.vasudevan commented on HBASE-18294: bq.I mean reducing the existing def values Yes as the case may be. > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-18294.01.patch, HBASE-18294.02.patch, > HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch, > HBASE-18294.06.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18127) Enable state to be passed between the region observer coprocessor hook calls
[ https://issues.apache.org/jira/browse/HBASE-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264980#comment-16264980 ] Abhishek Singh Chouhan commented on HBASE-18127: The cases that i saw had all the coproc hooks related to an operation being executed by a single thread. Do we have cases where some coproc hooks are executed by one thread and other by some other thread so we need to pass state between them? (need to check more here). For the general case where a thread from some pool executes the operation involving coproc hooks (all hooks related to that operation), i was thinking maybe we could have a util class that subclasses ThreadPoolExecutor and set the thread local in beforeExecute and remove in afterExecute, and then we use this threadpool whereever we're using threadpools for operations that involve coproc hooks. > Enable state to be passed between the region observer coprocessor hook calls > > > Key: HBASE-18127 > URL: https://issues.apache.org/jira/browse/HBASE-18127 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-18127.master.001.patch, > HBASE-18127.master.002.patch, HBASE-18127.master.002.patch, > HBASE-18127.master.003.patch, HBASE-18127.master.004.patch, > HBASE-18127.master.005.patch, HBASE-18127.master.005.patch, > HBASE-18127.master.006.patch > > > Allow regionobserver to optionally skip postPut/postDelete when > postBatchMutate was called. > Right now a RegionObserver can only statically implement one or the other. In > scenarios where we need to work sometimes on the single postPut and > postDelete hooks and sometimes on the batchMutate hooks, there is currently > no place to convey this information to the single hooks. I.e. the work has > been done in the batch, skip the single hooks. > There are various solutions: > 1. Allow some state to be passed _per operation_. > 2. Remove the single hooks and always only call batch hooks (with a default > wrapper for the single hooks). > 3. more? > [~apurtell], what we had discussed a few days back. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264979#comment-16264979 ] Anoop Sam John commented on HBASE-18294: bq.I think this is why the argument of increasing the existing default values comes. Because our decision metric has changed. Did not see Ram comment when I was writing. I mean reducing the existing def values. Correct Ram? Ya with that alone the issues what is noticed in tests can be solved, We noticed this as discussed in past. Just did not conclude then whether to reduce the def. Said that am still ok for on heap, continue with old way. > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-18294.01.patch, HBASE-18294.02.patch, > HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch, > HBASE-18294.06.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264978#comment-16264978 ] Anoop Sam John commented on HBASE-18294: Ya all that observation make sense. On heap tests u did. The better throughput may be because of less GC pauses too. We are clearing each of the region much earlier. We can achieve that same impact by reducing the current def 128MB flush size also. Just saying. The GC pause issue as well as the blocking updates issue (because of global barrier breach) can be addressed. As said, its ok even to continue the old way of check (heap size also check against 128MB) for on heap. If that is a concern. > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-18294.01.patch, HBASE-18294.02.patch, > HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch, > HBASE-18294.06.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264977#comment-16264977
]
binlijin commented on HBASE-19338:
--
bq. yscb put qps 8w(without patch) -> 9.8W(with patch)
9.8W(with patch ) looks like i am using asyncfs.
if i use multiwal and 1 wal and with the patch the qps is 11.2W.
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, HBASE-19338.master.001.patch,
> HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264974#comment-16264974 ] ramkrishna.s.vasudevan commented on HBASE-18294: bq.First, I noticed that master suffer major performance degradation w.r.t. branch-2 Can you file a JIRA for this if you have not already done. bq. We see that looking at the heap size instead of data size causes more disk flushes, since each store trigger flushes more frequently. However, the throughput increases significantly as we never reach global heap pressure. So when you say considering only the delta of the current patch - you are just trying to see per region flush as data + heap and so the global pressure reduces. Because we do more region level flushes. >From code one thing I noticed is that in the current master branch - when we >decide there is a lower mark breach for the memstore we decide based on both >data and heap size. (for onheap memstores). But the region that we select is >only by region data size. and I think that is why we are suggesting a change >there. So probably for onheap cases we should check (data + heap > 128MB) while selecting the best region and take it for flush. I think this is why the argument of increasing the existing default values comes. Because our decision metric has changed. > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-18294.01.patch, HBASE-18294.02.patch, > HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch, > HBASE-18294.06.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19323) Make netty engine default in hbase2
[ https://issues.apache.org/jira/browse/HBASE-19323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264973#comment-16264973 ] ramkrishna.s.vasudevan commented on HBASE-19323: bq.Netty's pool of BBs to read reqs is DBB only right? Yes. But I remember the netty server was not using our pool for sure. > Make netty engine default in hbase2 > --- > > Key: HBASE-19323 > URL: https://issues.apache.org/jira/browse/HBASE-19323 > Project: HBase > Issue Type: Task > Components: rpc >Reporter: stack > Fix For: 2.0.0-beta-1 > > Attachments: > 0001-HBASE-19323-Make-netty-engine-default-in-hbase2.patch, > HBASE-19323.master.001.patch > > > HBASE-17263 added netty rpc server. This issue is about making it default > given it has seen good service across two singles-days at scale. Netty > handles the scenario seen in HBASE-19320 (See tail of HBASE-19320 for > suggestion to netty the default) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18127) Enable state to be passed between the region observer coprocessor hook calls
[ https://issues.apache.org/jira/browse/HBASE-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264960#comment-16264960 ] ramkrishna.s.vasudevan edited comment on HBASE-18127 at 11/24/17 6:05 AM: -- How will the lifecycle be then (in case we need to hand over to diff thread pools that interacts with the CPs)? was (Author: ram_krish): How will the lifecycle be then? > Enable state to be passed between the region observer coprocessor hook calls > > > Key: HBASE-18127 > URL: https://issues.apache.org/jira/browse/HBASE-18127 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-18127.master.001.patch, > HBASE-18127.master.002.patch, HBASE-18127.master.002.patch, > HBASE-18127.master.003.patch, HBASE-18127.master.004.patch, > HBASE-18127.master.005.patch, HBASE-18127.master.005.patch, > HBASE-18127.master.006.patch > > > Allow regionobserver to optionally skip postPut/postDelete when > postBatchMutate was called. > Right now a RegionObserver can only statically implement one or the other. In > scenarios where we need to work sometimes on the single postPut and > postDelete hooks and sometimes on the batchMutate hooks, there is currently > no place to convey this information to the single hooks. I.e. the work has > been done in the batch, skip the single hooks. > There are various solutions: > 1. Allow some state to be passed _per operation_. > 2. Remove the single hooks and always only call batch hooks (with a default > wrapper for the single hooks). > 3. more? > [~apurtell], what we had discussed a few days back. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18127) Enable state to be passed between the region observer coprocessor hook calls
[ https://issues.apache.org/jira/browse/HBASE-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264960#comment-16264960 ] ramkrishna.s.vasudevan commented on HBASE-18127: How will the lifecycle be then? > Enable state to be passed between the region observer coprocessor hook calls > > > Key: HBASE-18127 > URL: https://issues.apache.org/jira/browse/HBASE-18127 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-18127.master.001.patch, > HBASE-18127.master.002.patch, HBASE-18127.master.002.patch, > HBASE-18127.master.003.patch, HBASE-18127.master.004.patch, > HBASE-18127.master.005.patch, HBASE-18127.master.005.patch, > HBASE-18127.master.006.patch > > > Allow regionobserver to optionally skip postPut/postDelete when > postBatchMutate was called. > Right now a RegionObserver can only statically implement one or the other. In > scenarios where we need to work sometimes on the single postPut and > postDelete hooks and sometimes on the batchMutate hooks, there is currently > no place to convey this information to the single hooks. I.e. the work has > been done in the batch, skip the single hooks. > There are various solutions: > 1. Allow some state to be passed _per operation_. > 2. Remove the single hooks and always only call batch hooks (with a default > wrapper for the single hooks). > 3. more? > [~apurtell], what we had discussed a few days back. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19323) Make netty engine default in hbase2
[ https://issues.apache.org/jira/browse/HBASE-19323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264957#comment-16264957 ] Anoop Sam John commented on HBASE-19323: That test was against old RpcServer in which version? May be test against latest SimpleRpcServer in branch-2 would be ideal. I did test taking alpha-4 release. On Netty Server, do we use the DBB pool that we have at RS side? (While reading reqs) At least for the response for read reqs, we have to make use as we have to make the cell block payload for PB response. So the simple RpcServer just uses same DBB pool for reading reqs also. Sorry we discussed abt this in that Netty issue I forgot now. Really sorry. Netty's pool of BBs to read reqs is DBB only right? > Make netty engine default in hbase2 > --- > > Key: HBASE-19323 > URL: https://issues.apache.org/jira/browse/HBASE-19323 > Project: HBase > Issue Type: Task > Components: rpc >Reporter: stack > Fix For: 2.0.0-beta-1 > > Attachments: > 0001-HBASE-19323-Make-netty-engine-default-in-hbase2.patch, > HBASE-19323.master.001.patch > > > HBASE-17263 added netty rpc server. This issue is about making it default > given it has seen good service across two singles-days at scale. Netty > handles the scenario seen in HBASE-19320 (See tail of HBASE-19320 for > suggestion to netty the default) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17319) Truncate table with preserve after split may cause truncation to fail
[
https://issues.apache.org/jira/browse/HBASE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264949#comment-16264949
]
Pankaj Kumar commented on HBASE-17319:
--
Hi [~allan163], we also facing this problem. Patch looks fine, any plan to
commit this fix?
> Truncate table with preserve after split may cause truncation to fail
> -
>
> Key: HBASE-17319
> URL: https://issues.apache.org/jira/browse/HBASE-17319
> Project: HBase
> Issue Type: Bug
> Components: Admin
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Allan Yang
>Assignee: Allan Yang
> Fix For: 2.0.0, 1.5.0
>
> Attachments: 17319.stack, HBASE-17319-branch-1.patch,
> HBASE-17319.patch, TestTruncateTableProcedure-output.tar.gz
>
>
> In truncateTableProcedue , when getting tables regions from meta to recreate
> new regions, split parents are not excluded, so the new regions can end up
> with the same start key, and the same region dir:
> {noformat}
> 2016-12-14 20:15:22,231 WARN [RegionOpenAndInitThread-writetest-1]
> regionserver.HRegionFileSystem: Trying to create a region that already exists
> on disk:
> hdfs://hbasedev1/zhengyan-hbase11-func2/.tmp/data/default/writetest/9b2c8d1539cd92661703ceb8a4d518a1
> {noformat}
> The truncateTableProcedue will retry forever and never get success.
> A attached unit test shows everything.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18127) Enable state to be passed between the region observer coprocessor hook calls
[ https://issues.apache.org/jira/browse/HBASE-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264947#comment-16264947 ] Anoop Sam John commented on HBASE-18127: Understood the problem/difficulty. But still when the flow handed over to other thread (from a pool or so), how to handle? U say explicit set on that and later reset once this work is over? We are facing this kind of ThreadLocal based issue in some other area/jira also. > Enable state to be passed between the region observer coprocessor hook calls > > > Key: HBASE-18127 > URL: https://issues.apache.org/jira/browse/HBASE-18127 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-18127.master.001.patch, > HBASE-18127.master.002.patch, HBASE-18127.master.002.patch, > HBASE-18127.master.003.patch, HBASE-18127.master.004.patch, > HBASE-18127.master.005.patch, HBASE-18127.master.005.patch, > HBASE-18127.master.006.patch > > > Allow regionobserver to optionally skip postPut/postDelete when > postBatchMutate was called. > Right now a RegionObserver can only statically implement one or the other. In > scenarios where we need to work sometimes on the single postPut and > postDelete hooks and sometimes on the batchMutate hooks, there is currently > no place to convey this information to the single hooks. I.e. the work has > been done in the batch, skip the single hooks. > There are various solutions: > 1. Allow some state to be passed _per operation_. > 2. Remove the single hooks and always only call batch hooks (with a default > wrapper for the single hooks). > 3. more? > [~apurtell], what we had discussed a few days back. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-19323) Make netty engine default in hbase2
[ https://issues.apache.org/jira/browse/HBASE-19323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264920#comment-16264920 ] Yu Li edited comment on HBASE-19323 at 11/24/17 5:47 AM: - bq. Do we need run some performance test for this? We already did a lot of performance test in HBASE-15756, please refer to [this comment|https://issues.apache.org/jira/browse/HBASE-15756?focusedCommentId=15743861=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15743861]. Please also note that netty rpc server uses much less threads to gain the same performance. Please let us know if you think the existing tests are not enough or have any more case to add. Thanks. was (Author: carp84): bq. Do we need run some performance test for this? We already did a lot performance test in HBASE-15756, see [this comment|https://issues.apache.org/jira/browse/HBASE-15756?focusedCommentId=15743861=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15743861]. Please also note that netty rpc server uses much less threads to meet the same performance. Please let us know if you think the existing test is not enough still more case to add. Thanks. > Make netty engine default in hbase2 > --- > > Key: HBASE-19323 > URL: https://issues.apache.org/jira/browse/HBASE-19323 > Project: HBase > Issue Type: Task > Components: rpc >Reporter: stack > Fix For: 2.0.0-beta-1 > > Attachments: > 0001-HBASE-19323-Make-netty-engine-default-in-hbase2.patch, > HBASE-19323.master.001.patch > > > HBASE-17263 added netty rpc server. This issue is about making it default > given it has seen good service across two singles-days at scale. Netty > handles the scenario seen in HBASE-19320 (See tail of HBASE-19320 for > suggestion to netty the default) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18309) Support multi threads in CleanerChore
[ https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-18309: -- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed into master and branch-2. Thanks [~reidchan] for the patch and thanks all for review. Closing issue. > Support multi threads in CleanerChore > - > > Key: HBASE-18309 > URL: https://issues.apache.org/jira/browse/HBASE-18309 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: Reid Chan > Fix For: 3.0.0, 2.0.0-beta-1 > > Attachments: HBASE-18309.master.001.patch, > HBASE-18309.master.002.patch, HBASE-18309.master.004.patch, > HBASE-18309.master.005.patch, HBASE-18309.master.006.patch, > HBASE-18309.master.007.patch, HBASE-18309.master.008.patch, > HBASE-18309.master.009.patch, HBASE-18309.master.010.patch, > HBASE-18309.master.011.patch, HBASE-18309.master.012.patch, > space_consumption_in_archive.png > > > There is only one thread in LogCleaner to clean oldWALs and in our big > cluster we find this is not enough. The number of files under oldWALs reach > the max-directory-items limit of HDFS and cause region server crash, so we > use multi threads for LogCleaner and the crash not happened any more. > What's more, currently there's only one thread iterating the archive > directory, and we could use multiple threads cleaning sub directories in > parallel to speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19334) User.runAsLoginUser not work in AccessController because it use a short circuited connection
[
https://issues.apache.org/jira/browse/HBASE-19334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264933#comment-16264933
]
Anoop Sam John commented on HBASE-19334:
Ah sorry. The CP path gets the user from RpcServer#getRequestUser() which is a
thread local based get. On reqs to other servers only the user is getting
passed from the connection user. Should we fix the actual problem than fixing
the AC case alone? I mean when short circuit connection is been used, the user
is changing depending on target server or which API is been used. Should we try
for a fix on that? Possible?
> User.runAsLoginUser not work in AccessController because it use a short
> circuited connection
>
>
> Key: HBASE-19334
> URL: https://issues.apache.org/jira/browse/HBASE-19334
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-19334.master.001.patch
>
>
> The short-circuited connection will bypass the RPC and the RPC context didn't
> change. So it still use the old RPC user to write ACL table and
> User.runAsLoginUser not work.
> AccessController's grant method.
> {code}
> User.runAsLoginUser(new PrivilegedExceptionAction() {
> @Override
> public Void run() throws Exception {
> // regionEnv is set at #start. Hopefully not null at this point.
> try (Table table = regionEnv.getConnection().
> getTable(AccessControlLists.ACL_TABLE_NAME)) {
>
> AccessControlLists.addUserPermission(regionEnv.getConfiguration(), perm,
> table,
> request.getMergeExistingPermissions());
> }
> return null;
> }
> });
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18127) Enable state to be passed between the region observer coprocessor hook calls
[ https://issues.apache.org/jira/browse/HBASE-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264932#comment-16264932 ] Abhishek Singh Chouhan commented on HBASE-18127: As Andrew pointed out we'd need to set this outside of RPC call context since coproc hooks are also called from other places that are not necessarily originating from a RPC call, eg. Flush table operation results in creation of sub procedure on the rs in which case the call context would be null. We might need to have a threadLocal in CoprocessorHost or Environment, however we'd need to set it and remove it across any threadpool doing any operations that involve coprocessor hooks (Doing this for rpc calls and procedures in general would cover most of the use cases but there might be more). [~anoop.hbase] [~ram_krish] [~apurtell] > Enable state to be passed between the region observer coprocessor hook calls > > > Key: HBASE-18127 > URL: https://issues.apache.org/jira/browse/HBASE-18127 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-18127.master.001.patch, > HBASE-18127.master.002.patch, HBASE-18127.master.002.patch, > HBASE-18127.master.003.patch, HBASE-18127.master.004.patch, > HBASE-18127.master.005.patch, HBASE-18127.master.005.patch, > HBASE-18127.master.006.patch > > > Allow regionobserver to optionally skip postPut/postDelete when > postBatchMutate was called. > Right now a RegionObserver can only statically implement one or the other. In > scenarios where we need to work sometimes on the single postPut and > postDelete hooks and sometimes on the batchMutate hooks, there is currently > no place to convey this information to the single hooks. I.e. the work has > been done in the batch, skip the single hooks. > There are various solutions: > 1. Allow some state to be passed _per operation_. > 2. Remove the single hooks and always only call batch hooks (with a default > wrapper for the single hooks). > 3. more? > [~apurtell], what we had discussed a few days back. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19336) Improve rsgroup to allow assign all tables within a specified namespace by only writing namespace
[
https://issues.apache.org/jira/browse/HBASE-19336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264928#comment-16264928
]
xinxin fan commented on HBASE-19336:
Thanks for review [~ted_yu]
Comand 'move_namespaces' just convert namespace into tables and finally
invoke function moveTables(tables, dest). Let me fix the last one.
> Improve rsgroup to allow assign all tables within a specified namespace by
> only writing namespace
> -
>
> Key: HBASE-19336
> URL: https://issues.apache.org/jira/browse/HBASE-19336
> Project: HBase
> Issue Type: Improvement
> Components: rsgroup
>Affects Versions: 2.0.0-alpha-4
>Reporter: xinxin fan
>Assignee: xinxin fan
> Attachments: HBASE-19336-master-V2.patch, HBASE-19336-master.patch
>
>
> Currently, use can only assign tables within a namespace from one group to
> another by writing all table names in move_tables_rsgroup command. Allowing
> to assign all tables within a specifed namespace by only wirting namespace
> name is useful.
> Usage as follows:
> {code:java}
> hbase(main):055:0> move_namespaces_rsgroup 'dest_rsgroup',['ns1']
> Took 2.2211 seconds
> {code}
> {code:java}
> hbase(main):051:0* move_servers_namespaces_rsgroup
> 'dest_rsgroup',['hbase39.lt.163.org:60020'],['ns1','ns2']
> Took 15.3710 seconds
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19323) Make netty engine default in hbase2
[ https://issues.apache.org/jira/browse/HBASE-19323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264920#comment-16264920 ] Yu Li commented on HBASE-19323: --- bq. Do we need run some performance test for this? We already did a lot performance test in HBASE-15756, see [this comment|https://issues.apache.org/jira/browse/HBASE-15756?focusedCommentId=15743861=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15743861]. Please also note that netty rpc server uses much less threads to meet the same performance. Please let us know if you think the existing test is not enough still more case to add. Thanks. > Make netty engine default in hbase2 > --- > > Key: HBASE-19323 > URL: https://issues.apache.org/jira/browse/HBASE-19323 > Project: HBase > Issue Type: Task > Components: rpc >Reporter: stack > Fix For: 2.0.0-beta-1 > > Attachments: > 0001-HBASE-19323-Make-netty-engine-default-in-hbase2.patch, > HBASE-19323.master.001.patch > > > HBASE-17263 added netty rpc server. This issue is about making it default > given it has seen good service across two singles-days at scale. Netty > handles the scenario seen in HBASE-19320 (See tail of HBASE-19320 for > suggestion to netty the default) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264898#comment-16264898
]
Ted Yu commented on HBASE-19338:
>From https://builds.apache.org/job/PreCommit-HBASE-Build/9994/console :
{code}
21:45:13 Modes: MultiJDK Jenkins Robot Docker ResetRepo UnitTests
21:45:13 Processing: HBASE-19338
21:45:13 ERROR: Unsure how to process HBASE-19338.
21:45:13
{code}
Looks like QA is taking a break.
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, HBASE-19338.master.001.patch,
> HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19300) TestMultithreadedTableMapper fails in branch-1.4
[
https://issues.apache.org/jira/browse/HBASE-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264895#comment-16264895
]
Ted Yu commented on HBASE-19300:
Looped TestMultithreadedTableMapper 10 times with patch which all passed.
> TestMultithreadedTableMapper fails in branch-1.4
>
>
> Key: HBASE-19300
> URL: https://issues.apache.org/jira/browse/HBASE-19300
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 19300.branch-1.4.patch, 19300.branch-1.4.patch
>
>
> From
> https://builds.apache.org/job/HBase-1.4/1023/jdk=JDK_1_7,label=Hadoop&&!H13/testReport/org.apache.hadoop.hbase.mapreduce/TestMultithreadedTableMapper/testMultithreadedTableMapper/
> :
> {code}
> java.lang.AssertionError
> at
> org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.verify(TestMultithreadedTableMapper.java:195)
> at
> org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.runTestOnTable(TestMultithreadedTableMapper.java:163)
> at
> org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.testMultithreadedTableMapper(TestMultithreadedTableMapper.java:136)
> {code}
> I ran the test locally which failed.
> Noticed the following in test output:
> {code}
> 2017-11-18 19:28:13,929 ERROR [hconnection-0x11db8653-shared--pool24-t9]
> protobuf.ResponseConverter(425): Results sent from server=703. But only got 0
> results completely atclient. Resetting the scanner to scan again.
> 2017-11-18 19:28:13,929 ERROR [hconnection-0x11db8653-shared--pool24-t3]
> protobuf.ResponseConverter(425): Results sent from server=703. But only got 0
> results completely atclient. Resetting the scanner to scan again.
> 2017-11-18 19:28:14,461 ERROR [hconnection-0x11db8653-shared--pool24-t8]
> protobuf.ResponseConverter(432): Exception while reading cells from
> result.Resetting the scanner toscan again.
> org.apache.hadoop.hbase.DoNotRetryIOException: Results sent from server=703.
> But only got 0 results completely at client. Resetting the scanner to scan
> again.
> at
> org.apache.hadoop.hbase.protobuf.ResponseConverter.getResults(ResponseConverter.java:426)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:284)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:388)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:362)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:142)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-18 19:28:14,464 ERROR [hconnection-0x11db8653-shared--pool24-t2]
> protobuf.ResponseConverter(432): Exception while reading cells from
> result.Resetting the scanner toscan again.
> java.io.EOFException: Partial cell read
> at
> org.apache.hadoop.hbase.codec.BaseDecoder.rethrowEofException(BaseDecoder.java:86)
> at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:70)
> at
> org.apache.hadoop.hbase.protobuf.ResponseConverter.getResults(ResponseConverter.java:419)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:284)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:388)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:362)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:142)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Premature EOF from inputStream
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:202)
> at
[jira] [Commented] (HBASE-19300) TestMultithreadedTableMapper fails in branch-1.4
[
https://issues.apache.org/jira/browse/HBASE-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264882#comment-16264882
]
Ted Yu commented on HBASE-19300:
https://builds.apache.org/job/PreCommit-HBASE-Build/9996/console :
{code}
22:46:13 Modes: MultiJDK Jenkins Robot Docker ResetRepo UnitTests
22:46:13 Processing: HBASE-19300
22:46:13 ERROR: Unsure how to process HBASE-19300.
{code}
Not sure whether patch filename was not well formed or something else.
> TestMultithreadedTableMapper fails in branch-1.4
>
>
> Key: HBASE-19300
> URL: https://issues.apache.org/jira/browse/HBASE-19300
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 19300.branch-1.4.patch, 19300.branch-1.4.patch
>
>
> From
> https://builds.apache.org/job/HBase-1.4/1023/jdk=JDK_1_7,label=Hadoop&&!H13/testReport/org.apache.hadoop.hbase.mapreduce/TestMultithreadedTableMapper/testMultithreadedTableMapper/
> :
> {code}
> java.lang.AssertionError
> at
> org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.verify(TestMultithreadedTableMapper.java:195)
> at
> org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.runTestOnTable(TestMultithreadedTableMapper.java:163)
> at
> org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.testMultithreadedTableMapper(TestMultithreadedTableMapper.java:136)
> {code}
> I ran the test locally which failed.
> Noticed the following in test output:
> {code}
> 2017-11-18 19:28:13,929 ERROR [hconnection-0x11db8653-shared--pool24-t9]
> protobuf.ResponseConverter(425): Results sent from server=703. But only got 0
> results completely atclient. Resetting the scanner to scan again.
> 2017-11-18 19:28:13,929 ERROR [hconnection-0x11db8653-shared--pool24-t3]
> protobuf.ResponseConverter(425): Results sent from server=703. But only got 0
> results completely atclient. Resetting the scanner to scan again.
> 2017-11-18 19:28:14,461 ERROR [hconnection-0x11db8653-shared--pool24-t8]
> protobuf.ResponseConverter(432): Exception while reading cells from
> result.Resetting the scanner toscan again.
> org.apache.hadoop.hbase.DoNotRetryIOException: Results sent from server=703.
> But only got 0 results completely at client. Resetting the scanner to scan
> again.
> at
> org.apache.hadoop.hbase.protobuf.ResponseConverter.getResults(ResponseConverter.java:426)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:284)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:388)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:362)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:142)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-18 19:28:14,464 ERROR [hconnection-0x11db8653-shared--pool24-t2]
> protobuf.ResponseConverter(432): Exception while reading cells from
> result.Resetting the scanner toscan again.
> java.io.EOFException: Partial cell read
> at
> org.apache.hadoop.hbase.codec.BaseDecoder.rethrowEofException(BaseDecoder.java:86)
> at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:70)
> at
> org.apache.hadoop.hbase.protobuf.ResponseConverter.getResults(ResponseConverter.java:419)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:284)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:388)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:362)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:142)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
[jira] [Commented] (HBASE-19336) Improve rsgroup to allow assign all tables within a specified namespace by only writing namespace
[
https://issues.apache.org/jira/browse/HBASE-19336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264872#comment-16264872
]
Ted Yu commented on HBASE-19336:
{code}
+def move_namespaces(dest, *args)
+ tables = java.util.HashSet.new
{code}
Variable name should match what the command is moving.
{code}
+ servers = java.util.HashSet.new
+ tables = java.util.HashSet.new
{code}
Same with tables above.
{code}
+Reassign tables from one RegionServer group to another.
{code}
tables -> namespaces
> Improve rsgroup to allow assign all tables within a specified namespace by
> only writing namespace
> -
>
> Key: HBASE-19336
> URL: https://issues.apache.org/jira/browse/HBASE-19336
> Project: HBase
> Issue Type: Improvement
> Components: rsgroup
>Affects Versions: 2.0.0-alpha-4
>Reporter: xinxin fan
>Assignee: xinxin fan
> Attachments: HBASE-19336-master-V2.patch, HBASE-19336-master.patch
>
>
> Currently, use can only assign tables within a namespace from one group to
> another by writing all table names in move_tables_rsgroup command. Allowing
> to assign all tables within a specifed namespace by only wirting namespace
> name is useful.
> Usage as follows:
> {code:java}
> hbase(main):055:0> move_namespaces_rsgroup 'dest_rsgroup',['ns1']
> Took 2.2211 seconds
> {code}
> {code:java}
> hbase(main):051:0* move_servers_namespaces_rsgroup
> 'dest_rsgroup',['hbase39.lt.163.org:60020'],['ns1','ns2']
> Took 15.3710 seconds
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19336) Improve rsgroup to allow assign all tables within a specified namespace by only writing namespace
[
https://issues.apache.org/jira/browse/HBASE-19336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264870#comment-16264870
]
xinxin fan commented on HBASE-19336:
In HBASE-19336-master-V2.patch, just add two command 'move_namespaces_rsgroup'
and 'move_servers_namespaces_rsgroup' to assign all tables within specified
namespaces to dest rsgroup
> Improve rsgroup to allow assign all tables within a specified namespace by
> only writing namespace
> -
>
> Key: HBASE-19336
> URL: https://issues.apache.org/jira/browse/HBASE-19336
> Project: HBase
> Issue Type: Improvement
> Components: rsgroup
>Affects Versions: 2.0.0-alpha-4
>Reporter: xinxin fan
>Assignee: xinxin fan
> Attachments: HBASE-19336-master-V2.patch, HBASE-19336-master.patch
>
>
> Currently, use can only assign tables within a namespace from one group to
> another by writing all table names in move_tables_rsgroup command. Allowing
> to assign all tables within a specifed namespace by only wirting namespace
> name is useful.
> Usage as follows:
> {code:java}
> hbase(main):055:0> move_namespaces_rsgroup 'dest_rsgroup',['ns1']
> Took 2.2211 seconds
> {code}
> {code:java}
> hbase(main):051:0* move_servers_namespaces_rsgroup
> 'dest_rsgroup',['hbase39.lt.163.org:60020'],['ns1','ns2']
> Took 15.3710 seconds
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19336) Improve rsgroup to allow assign all tables within a specified namespace by only writing namespace
[
https://issues.apache.org/jira/browse/HBASE-19336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xinxin fan updated HBASE-19336:
---
Description:
Currently, use can only assign tables within a namespace from one group to
another by writing all table names in move_tables_rsgroup command. Allowing to
assign all tables within a specifed namespace by only wirting namespace name is
useful.
Usage as follows:
{code:java}
hbase(main):055:0> move_namespaces_rsgroup 'dest_rsgroup',['ns1']
Took 2.2211 seconds
{code}
{code:java}
hbase(main):051:0* move_servers_namespaces_rsgroup
'dest_rsgroup',['hbase39.lt.163.org:60020'],['ns1','ns2']
Took 15.3710 seconds
{code}
was:
Currently, use can only assign tables within a namespace from one group to
another by writing all table names in move_tables_rsgroup command. Allowing to
assign all tables within a specifed namespace by only wirting namespace name is
useful.
Usage as follows:
{code:java}
hbase(main):055:0> move_tables_rsgroup 'default',['@ns1']
Took 2.2211 seconds
{code}
{code:java}
hbase(main):051:0* move_servers_tables_rsgroup
'rsgroup1',['hbase39.lt.163.org:60020'],['@ns1','@ns2','table3']
Took 15.3710 seconds
{code}
> Improve rsgroup to allow assign all tables within a specified namespace by
> only writing namespace
> -
>
> Key: HBASE-19336
> URL: https://issues.apache.org/jira/browse/HBASE-19336
> Project: HBase
> Issue Type: Improvement
> Components: rsgroup
>Affects Versions: 2.0.0-alpha-4
>Reporter: xinxin fan
>Assignee: xinxin fan
> Attachments: HBASE-19336-master-V2.patch, HBASE-19336-master.patch
>
>
> Currently, use can only assign tables within a namespace from one group to
> another by writing all table names in move_tables_rsgroup command. Allowing
> to assign all tables within a specifed namespace by only wirting namespace
> name is useful.
> Usage as follows:
> {code:java}
> hbase(main):055:0> move_namespaces_rsgroup 'dest_rsgroup',['ns1']
> Took 2.2211 seconds
> {code}
> {code:java}
> hbase(main):051:0* move_servers_namespaces_rsgroup
> 'dest_rsgroup',['hbase39.lt.163.org:60020'],['ns1','ns2']
> Took 15.3710 seconds
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19336) Improve rsgroup to allow assign all tables within a specified namespace by only writing namespace
[
https://issues.apache.org/jira/browse/HBASE-19336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xinxin fan updated HBASE-19336:
---
Attachment: HBASE-19336-master-V2.patch
> Improve rsgroup to allow assign all tables within a specified namespace by
> only writing namespace
> -
>
> Key: HBASE-19336
> URL: https://issues.apache.org/jira/browse/HBASE-19336
> Project: HBase
> Issue Type: Improvement
> Components: rsgroup
>Affects Versions: 2.0.0-alpha-4
>Reporter: xinxin fan
>Assignee: xinxin fan
> Attachments: HBASE-19336-master-V2.patch, HBASE-19336-master.patch
>
>
> Currently, use can only assign tables within a namespace from one group to
> another by writing all table names in move_tables_rsgroup command. Allowing
> to assign all tables within a specifed namespace by only wirting namespace
> name is useful.
> Usage as follows:
> {code:java}
> hbase(main):055:0> move_tables_rsgroup 'default',['@ns1']
> Took 2.2211 seconds
> {code}
> {code:java}
> hbase(main):051:0* move_servers_tables_rsgroup
> 'rsgroup1',['hbase39.lt.163.org:60020'],['@ns1','@ns2','table3']
> Took 15.3710 seconds
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264850#comment-16264850
]
Duo Zhang commented on HBASE-19338:
---
{quote}
Yes, User.getCurrent throws IOException, looks like we can not use orElseGet
here.
{quote}
OK, that must be why I do not use User::getCurrent at the first place.
Nice finding. Maybe we could introduce a helper method to avoid writing the if
condition in multiple places. But this is not a critical problem, can do it
later, or just ignore it is also fine.
+1 on the patch.
Thanks.
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, HBASE-19338.master.001.patch,
> HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264848#comment-16264848
]
binlijin edited comment on HBASE-19338 at 11/24/17 12:38 AM:
-
bq. Does User.getCurrent throw any exceptions? If no, we can use
orElseGet(User::getCurrent), else the only solution is to write the if
condition explicitly like the patch here.
Yes, User.getCurrent throws IOException, looks like we can not use orElseGet
here.
was (Author: aoxiang):
bp. Does User.getCurrent throw any exceptions? If no, we can use
orElseGet(User::getCurrent), else the only solution is to write the if
condition explicitly like the patch here.
Yes, User.getCurrent throws IOException, looks like we can not use orElseGet
here.
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, HBASE-19338.master.001.patch,
> HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264848#comment-16264848
]
binlijin commented on HBASE-19338:
--
bp. Does User.getCurrent throw any exceptions? If no, we can use
orElseGet(User::getCurrent), else the only solution is to write the if
condition explicitly like the patch here.
Yes, User.getCurrent throws IOException, looks like we can not use orElseGet
here.
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, HBASE-19338.master.001.patch,
> HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19334) User.runAsLoginUser not work in AccessController because it use a short circuited connection
[
https://issues.apache.org/jira/browse/HBASE-19334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264841#comment-16264841
]
Guanghao Zhang commented on HBASE-19334:
bq, Now the code is using new getConnection() which reuses the initial cluster
connection (short circuited) in RS. For this the user is always the super user
who started RS process.
I thought you misunderstand what I said. The user was stored by a threadlocal
variable. If a user A call grant method and then use short curcuited connection
to bypass rpc. Then the user still is user A. So User.runAsLoginUser not work.
You can try replace put(List) API with put(Put) and run TestAccessControl* test
to see the result.
> User.runAsLoginUser not work in AccessController because it use a short
> circuited connection
>
>
> Key: HBASE-19334
> URL: https://issues.apache.org/jira/browse/HBASE-19334
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-19334.master.001.patch
>
>
> The short-circuited connection will bypass the RPC and the RPC context didn't
> change. So it still use the old RPC user to write ACL table and
> User.runAsLoginUser not work.
> AccessController's grant method.
> {code}
> User.runAsLoginUser(new PrivilegedExceptionAction() {
> @Override
> public Void run() throws Exception {
> // regionEnv is set at #start. Hopefully not null at this point.
> try (Table table = regionEnv.getConnection().
> getTable(AccessControlLists.ACL_TABLE_NAME)) {
>
> AccessControlLists.addUserPermission(regionEnv.getConfiguration(), perm,
> table,
> request.getMergeExistingPermissions());
> }
> return null;
> }
> });
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264826#comment-16264826
]
Duo Zhang commented on HBASE-19338:
---
Does User.getCurrent throw any exceptions? If no, we can use
orElseGet(User::getCurrent), else the only solution is to write the if
condition explicitly like the patch here.
Thanks.
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, HBASE-19338.master.001.patch,
> HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264812#comment-16264812
]
Ted Yu commented on HBASE-19338:
Since User / UserProvider doesn't implement Supplier, we cannot use the method
directly.
I suggest going with the tested formation.
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, HBASE-19338.master.001.patch,
> HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264808#comment-16264808
]
Chia-Ping Tsai commented on HBASE-19338:
Could we use
[orElseGet|https://docs.oracle.com/javase/8/docs/api/java/util/Optional.html#orElseGet-java.util.function.Supplier-]
to rewrite the patch? It is simpler and more graceful I think.
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, HBASE-19338.master.001.patch,
> HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19300) TestMultithreadedTableMapper fails in branch-1.4
[
https://issues.apache.org/jira/browse/HBASE-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-19300:
---
Attachment: 19300.branch-1.4.patch
> TestMultithreadedTableMapper fails in branch-1.4
>
>
> Key: HBASE-19300
> URL: https://issues.apache.org/jira/browse/HBASE-19300
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 19300.branch-1.4.patch, 19300.branch-1.4.patch
>
>
> From
> https://builds.apache.org/job/HBase-1.4/1023/jdk=JDK_1_7,label=Hadoop&&!H13/testReport/org.apache.hadoop.hbase.mapreduce/TestMultithreadedTableMapper/testMultithreadedTableMapper/
> :
> {code}
> java.lang.AssertionError
> at
> org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.verify(TestMultithreadedTableMapper.java:195)
> at
> org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.runTestOnTable(TestMultithreadedTableMapper.java:163)
> at
> org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.testMultithreadedTableMapper(TestMultithreadedTableMapper.java:136)
> {code}
> I ran the test locally which failed.
> Noticed the following in test output:
> {code}
> 2017-11-18 19:28:13,929 ERROR [hconnection-0x11db8653-shared--pool24-t9]
> protobuf.ResponseConverter(425): Results sent from server=703. But only got 0
> results completely atclient. Resetting the scanner to scan again.
> 2017-11-18 19:28:13,929 ERROR [hconnection-0x11db8653-shared--pool24-t3]
> protobuf.ResponseConverter(425): Results sent from server=703. But only got 0
> results completely atclient. Resetting the scanner to scan again.
> 2017-11-18 19:28:14,461 ERROR [hconnection-0x11db8653-shared--pool24-t8]
> protobuf.ResponseConverter(432): Exception while reading cells from
> result.Resetting the scanner toscan again.
> org.apache.hadoop.hbase.DoNotRetryIOException: Results sent from server=703.
> But only got 0 results completely at client. Resetting the scanner to scan
> again.
> at
> org.apache.hadoop.hbase.protobuf.ResponseConverter.getResults(ResponseConverter.java:426)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:284)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:388)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:362)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:142)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-18 19:28:14,464 ERROR [hconnection-0x11db8653-shared--pool24-t2]
> protobuf.ResponseConverter(432): Exception while reading cells from
> result.Resetting the scanner toscan again.
> java.io.EOFException: Partial cell read
> at
> org.apache.hadoop.hbase.codec.BaseDecoder.rethrowEofException(BaseDecoder.java:86)
> at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:70)
> at
> org.apache.hadoop.hbase.protobuf.ResponseConverter.getResults(ResponseConverter.java:419)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:284)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:388)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:362)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:142)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Premature EOF from inputStream
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:202)
> at org.apache.hadoop.hbase.KeyValueUtil.iscreate(KeyValueUtil.java:611)
> at
>
[jira] [Updated] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-19338:
---
Attachment: 19338.master.003.patch
Patch v3 fixes the similar call in VisibilityUtils and AccessController since
they are on hot path.
Lijin's patches didn't carry header. So the format stays the same.
Whoever commits the final patch, please give Lijin credit.
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: 19338.master.003.patch, HBASE-19338.master.001.patch,
> HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264742#comment-16264742
]
Anoop Sam John commented on HBASE-19338:
I was checking code for similar usages. Found places like
SecureBulkLoadManager, VisibilityUtils, AccessController#getActiveUser,
Export. Ya AC and VC might be worth fixing. Another issue also fine
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: HBASE-19338.master.001.patch,
> HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19323) Make netty engine default in hbase2
[ https://issues.apache.org/jira/browse/HBASE-19323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264741#comment-16264741 ] Anoop Sam John commented on HBASE-19323: Ya that will be good. I did a quick run with PE random write test on a single node cluster. This writes 25 GB data with 100 threads. The perf look almost similar with SimpleRpcServer is having just 1% adv. This is not a large test. Just did a quick one to know the initial values. > Make netty engine default in hbase2 > --- > > Key: HBASE-19323 > URL: https://issues.apache.org/jira/browse/HBASE-19323 > Project: HBase > Issue Type: Task > Components: rpc >Reporter: stack > Fix For: 2.0.0-beta-1 > > Attachments: > 0001-HBASE-19323-Make-netty-engine-default-in-hbase2.patch, > HBASE-19323.master.001.patch > > > HBASE-17263 added netty rpc server. This issue is about making it default > given it has seen good service across two singles-days at scale. Netty > handles the scenario seen in HBASE-19320 (See tail of HBASE-19320 for > suggestion to netty the default) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264737#comment-16264737
]
Chia-Ping Tsai commented on HBASE-19338:
+1
The root cause is the slow {{User#getCurrent}}? If so, could you fix the
similar bad usage in code base?
{code:title=VisibilityUtils.class}
public static User getActiveUser() throws IOException {
User user = RpcServer.getRequestUser().orElse(User.getCurrent());
if (LOG.isTraceEnabled()) {
LOG.trace("Current active user name is " + user.getShortName());
}
return user;
}
{code}
Or we can file another jira to fix it.
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: HBASE-19338.master.001.patch,
> HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264724#comment-16264724
]
Anoop Sam John commented on HBASE-19338:
+1
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: HBASE-19338.master.001.patch,
> HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19266) TestAcidGuarantees should cover adaptive in-memory compaction
[ https://issues.apache.org/jira/browse/HBASE-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264717#comment-16264717 ] Hudson commented on HBASE-19266: FAILURE: Integrated in Jenkins build HBase-2.0 #903 (See [https://builds.apache.org/job/HBase-2.0/903/]) HBASE-19266 TestAcidGuarantees should cover adaptive in-memory (chia7712: rev 66e650c39b7d794e9cd774add7b9f2be9d064e8f) * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/TestAcidGuaranteesWithEagerPolicy.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/TestAcidGuaranteesWithNoInMemCompaction.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/AcidGuaranteesTestTool.java * (delete) hbase-server/src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/TestAcidGuaranteesWithAdaptivePolicy.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/TestAcidGuaranteesWithBasicPolicy.java > TestAcidGuarantees should cover adaptive in-memory compaction > - > > Key: HBASE-19266 > URL: https://issues.apache.org/jira/browse/HBASE-19266 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19266.v0.patch, HBASE-19266.v1.patch > > > Currently TestAcidGuarantees populates 3 policies of (in-memory) compaction. > Adaptive in-memory compaction is new and should be added as 4th compaction > policy. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19334) User.runAsLoginUser not work in AccessController because it use a short circuited connection
[
https://issues.apache.org/jira/browse/HBASE-19334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264685#comment-16264685
]
Anoop Sam John commented on HBASE-19334:
I believe u added that TODO abt why using List instead of Put based API
before the changes/cleanup that happened around CP. Previously we had
getTable() in CoprocessorEnvironment which will create a short circuited
connection for the table then. So if u see the old code, u can see that every
time the getTable was called, we created new connection and the user been
passed as the current user context from which the CP is getting executed. So
the issue with the asyn process or not came in. Now the code is using new
getConnection() which reuses the initial cluster connection (short circuited)
in RS. For this the user is always the super user who started RS process. Said
that the TODO can be fulfilled with out any change as of now. Just replace
put(List) API with put(Put). I did not test this but I believe this will
happen.
Now this raised many Qs
1. In a secure cluster, the getConnection() might not be the correct one to be
used by CPs. As there is no way to pass the identity. Forget abt the short
circuited connection to same server. Even if the APIs used over the connection
is targeting another server, there the user will be the hbase super user. The
new createConnection() do not have such an issue and that looks to be the
equivalent of the old way. Any way in the doc addendum patch for the other
issue (createCOnnection), I have added a doc to getConnection() API that the
user will be super user. Pls see if we need stronger message, we can add.
2. The original issue of usage of diff APIs based on AsynProcess used or not -
We have to think abt this also. Because this will be applicable with the new
connection created using createConnection() API. The RpcContext() not getting
passed for the one set of APIs will be an issue at some point.
What do u all think? [~zghaobac], [~stack]
> User.runAsLoginUser not work in AccessController because it use a short
> circuited connection
>
>
> Key: HBASE-19334
> URL: https://issues.apache.org/jira/browse/HBASE-19334
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-19334.master.001.patch
>
>
> The short-circuited connection will bypass the RPC and the RPC context didn't
> change. So it still use the old RPC user to write ACL table and
> User.runAsLoginUser not work.
> AccessController's grant method.
> {code}
> User.runAsLoginUser(new PrivilegedExceptionAction() {
> @Override
> public Void run() throws Exception {
> // regionEnv is set at #start. Hopefully not null at this point.
> try (Table table = regionEnv.getConnection().
> getTable(AccessControlLists.ACL_TABLE_NAME)) {
>
> AccessControlLists.addUserPermission(regionEnv.getConfiguration(), perm,
> table,
> request.getMergeExistingPermissions());
> }
> return null;
> }
> });
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19163) "Maximum lock count exceeded" from region server's batch processing
[
https://issues.apache.org/jira/browse/HBASE-19163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264682#comment-16264682
]
huaxiang sun commented on HBASE-19163:
--
Yes, [~saint@gmail.com]. The issue is filed, HBASE-19308. The error is
logged out of this method. I will check to see if it is logged by other
callers, thanks.
> "Maximum lock count exceeded" from region server's batch processing
> ---
>
> Key: HBASE-19163
> URL: https://issues.apache.org/jira/browse/HBASE-19163
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 3.0.0, 1.2.7, 2.0.0-alpha-3
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-19163-master-v001.patch,
> HBASE-19163.master.001.patch, HBASE-19163.master.002.patch,
> HBASE-19163.master.004.patch, HBASE-19163.master.005.patch,
> HBASE-19163.master.006.patch, unittest-case.diff
>
>
> In one of use cases, we found the following exception and replication is
> stuck.
> {code}
> 2017-10-25 19:41:17,199 WARN [hconnection-0x28db294f-shared--pool4-t936]
> client.AsyncProcess: #3, table=foo, attempt=5/5 failed=262836ops, last
> exception: java.io.IOException: java.io.IOException: Maximum lock count
> exceeded
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2215)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
> Caused by: java.lang.Error: Maximum lock count exceeded
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.fullTryAcquireShared(ReentrantReadWriteLock.java:528)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared(ReentrantReadWriteLock.java:488)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1327)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:871)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLock(HRegion.java:5163)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3018)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2877)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:753)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:715)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2148)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
> ... 3 more
> {code}
> While we are still examining the data pattern, it is sure that there are too
> many mutations in the batch against the same row, this exceeds the maximum
> 64k shared lock count and it throws an error and failed the whole batch.
> There are two approaches to solve this issue.
> 1). Let's say there are mutations against the same row in the batch, we just
> need to acquire the lock once for the same row vs to acquire the lock for
> each mutation.
> 2). We catch the error and start to process whatever it gets and loop back.
> With HBASE-17924, approach 1 seems easy to implement now.
> Create the jira and will post update/patch when investigation moving forward.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19093) Check Admin/Table to ensure all operations go via AccessControl
[ https://issues.apache.org/jira/browse/HBASE-19093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264668#comment-16264668 ] Anoop Sam John commented on HBASE-19093: Ya that is what.. am not at all sure how we can a real test case for this. This issue, when raised, the intent is to have a UT or check for each of the Admin/Table methods to make sure ACL been checked for each of them? Ya a UT if possible is really good. But if that is really tough (I think so), better we dont have. WDYT? Lets check each of the API reading code and fix issues if found any. Lets open sub task for each of the issue. Sounds ok? > Check Admin/Table to ensure all operations go via AccessControl > --- > > Key: HBASE-19093 > URL: https://issues.apache.org/jira/browse/HBASE-19093 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Balazs Meszaros >Priority: Blocker > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19093.master.001.patch, > HBASE-19093.master.002.patch, RegionObserver.txt > > > A cursory review of Admin Interface has a bunch of methods as open, with out > AccessControl checks. For example, procedure executor has not check on it. > This issue is about given the Admin and Table Interfaces a once-over to see > what is missing and to fill in access control where missing. > This is a follow-on from work over in HBASE-19048 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[ https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264641#comment-16264641 ] ramkrishna.s.vasudevan commented on HBASE-19092: [~anoop.hbase] Do you want to have a look at this before we commit it? > Make Tag IA.LimitedPrivate and expose for CPs > - > > Key: HBASE-19092 > URL: https://issues.apache.org/jira/browse/HBASE-19092 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19092-branch-2.patch, > HBASE-19092-branch-2_5.patch, HBASE-19092-branch-2_5.patch, > HBASE-19092.branch-2.0.02.patch, HBASE-19092_001-branch-2.patch, > HBASE-19092_001.patch, HBASE-19092_002-branch-2.patch, HBASE-19092_002.patch, > HBASE-19092_004.patch, HBASE-19092_3.patch, HBASE-19092_4.patch > > > We need to make tags as LimitedPrivate as some use cases are trying to use > tags like timeline server. The same topic was discussed in dev@ and also in > HBASE-18995. > Shall we target this for beta1 - cc [~saint@gmail.com]. > So once we do this all related Util methods and APIs should also move to > LimitedPrivate Util classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264595#comment-16264595
]
Hadoop QA commented on HBASE-19338:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
48s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
39s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
53m 30s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 83m
29s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}159m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19338 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12899063/HBASE-19338.master.002.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 1e289d358cb5 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 3e2941a49e |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/9992/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/9992/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> Performance regression in
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264574#comment-16264574
]
Ted Yu commented on HBASE-19338:
This seems to be introduced by HBASE-18878
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: HBASE-19338.master.001.patch,
> HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19266) TestAcidGuarantees should cover adaptive in-memory compaction
[ https://issues.apache.org/jira/browse/HBASE-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19266: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Push to branch-2 and master. Thanks for the review. [~tedyu] > TestAcidGuarantees should cover adaptive in-memory compaction > - > > Key: HBASE-19266 > URL: https://issues.apache.org/jira/browse/HBASE-19266 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19266.v0.patch, HBASE-19266.v1.patch > > > Currently TestAcidGuarantees populates 3 policies of (in-memory) compaction. > Adaptive in-memory compaction is new and should be added as 4th compaction > policy. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19093) Check Admin/Table to ensure all operations go via AccessControl
[
https://issues.apache.org/jira/browse/HBASE-19093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264484#comment-16264484
]
Balazs Meszaros commented on HBASE-19093:
-
I have started to modify the test, where I check the methods of
{{MasterService.Interface}}. It is possible to find theirs corresponding
methods in {{AccessController}}, but the names of some methods are different
(e.g. {{moveRegion}} -> {{preMove}}, {{assignRegion}} -> {{preAssign}},
{{addColumn}} -> {{modifyTable}}, ...). I can set up some rules for these
cases, but the test will be a little bit more complicated. This approach seems
to be better, because for example {{setBalancerRunning}} does not call
{{MasterObserver}} at all.
> Check Admin/Table to ensure all operations go via AccessControl
> ---
>
> Key: HBASE-19093
> URL: https://issues.apache.org/jira/browse/HBASE-19093
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Assignee: Balazs Meszaros
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19093.master.001.patch,
> HBASE-19093.master.002.patch, RegionObserver.txt
>
>
> A cursory review of Admin Interface has a bunch of methods as open, with out
> AccessControl checks. For example, procedure executor has not check on it.
> This issue is about given the Admin and Table Interfaces a once-over to see
> what is missing and to fill in access control where missing.
> This is a follow-on from work over in HBASE-19048
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19266) TestAcidGuarantees should cover adaptive in-memory compaction
[ https://issues.apache.org/jira/browse/HBASE-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264462#comment-16264462 ] Chia-Ping Tsai commented on HBASE-19266: HBASE-19339 > TestAcidGuarantees should cover adaptive in-memory compaction > - > > Key: HBASE-19266 > URL: https://issues.apache.org/jira/browse/HBASE-19266 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19266.v0.patch, HBASE-19266.v1.patch > > > Currently TestAcidGuarantees populates 3 policies of (in-memory) compaction. > Adaptive in-memory compaction is new and should be added as 4th compaction > policy. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19339) Enable TestAcidGuaranteesWithEagerPolicy and TestAcidGuaranteesWithAdaptivePolicy
Chia-Ping Tsai created HBASE-19339: -- Summary: Enable TestAcidGuaranteesWithEagerPolicy and TestAcidGuaranteesWithAdaptivePolicy Key: HBASE-19339 URL: https://issues.apache.org/jira/browse/HBASE-19339 Project: HBase Issue Type: Task Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai It is a follow-up of HBASE-19266. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264459#comment-16264459 ] Eshcar Hillel commented on HBASE-18294: --- I am back with some numbers. First, I noticed that master suffer major performance degradation w.r.t. branch-2. This is out of the scope of this Jira and I plan to discuss this issue separately. Considering only the delta presented in the current patch here is what I observe for write-only workload with default parameters (Basic memstore compaction) ||code||Throughput||#flushes||#global heap pressure log lines|| |master|58-59Kops|~1250|~700| |master+patch|70-71Kops|~2000|0| And we see similar trends when running with no memstore compaction. We see that looking at the heap size instead of data size causes more disk flushes, since each store trigger flushes more frequently. However, the throughput increases significantly as we *never* reach global heap pressure. IMO this demonstrates that frequent pressure due to global heap size is not healthy, at least from performance perspective. These experiments show the benefit of the patch for on-heap stores. I think it is best to enforce symmetric behavior for on-heap and off-heap stores. And this should start with the naming convention. So let's not have data size vs. on-heap size but rather on-heap vs off-heap size. The reason I think we should have two (optional) threshold is that the space allocated on- and off-heap and their usage can vary. Or let me phrase it as a question: is there a reason not to let the admin the liberty to set these threshold differently?? if they are not set by the admin they get the default value (which is currently 128MB). > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-18294.01.patch, HBASE-18294.02.patch, > HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch, > HBASE-18294.06.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19266) TestAcidGuarantees should cover adaptive in-memory compaction
[ https://issues.apache.org/jira/browse/HBASE-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264458#comment-16264458 ] Chia-Ping Tsai commented on HBASE-19266: bq. I maintain my previous stance. Once the 'memstoreSize to a negative value' error is eliminated, we can enable these two tests. Let me log it in another jira. > TestAcidGuarantees should cover adaptive in-memory compaction > - > > Key: HBASE-19266 > URL: https://issues.apache.org/jira/browse/HBASE-19266 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19266.v0.patch, HBASE-19266.v1.patch > > > Currently TestAcidGuarantees populates 3 policies of (in-memory) compaction. > Adaptive in-memory compaction is new and should be added as 4th compaction > policy. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19266) TestAcidGuarantees should cover adaptive in-memory compaction
[ https://issues.apache.org/jira/browse/HBASE-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264455#comment-16264455 ] Ted Yu commented on HBASE-19266: bq. Agree to disable eager and adaptive for now. I maintain my previous stance. Once the 'memstoreSize to a negative value' error is eliminated, we can enable these two tests. > TestAcidGuarantees should cover adaptive in-memory compaction > - > > Key: HBASE-19266 > URL: https://issues.apache.org/jira/browse/HBASE-19266 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19266.v0.patch, HBASE-19266.v1.patch > > > Currently TestAcidGuarantees populates 3 policies of (in-memory) compaction. > Adaptive in-memory compaction is new and should be added as 4th compaction > policy. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19266) TestAcidGuarantees should cover adaptive in-memory compaction
[ https://issues.apache.org/jira/browse/HBASE-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19266: --- Fix Version/s: 2.0.0-beta-1 > TestAcidGuarantees should cover adaptive in-memory compaction > - > > Key: HBASE-19266 > URL: https://issues.apache.org/jira/browse/HBASE-19266 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19266.v0.patch, HBASE-19266.v1.patch > > > Currently TestAcidGuarantees populates 3 policies of (in-memory) compaction. > Adaptive in-memory compaction is new and should be added as 4th compaction > policy. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19266) TestAcidGuarantees should cover adaptive in-memory compaction
[ https://issues.apache.org/jira/browse/HBASE-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264453#comment-16264453 ] Chia-Ping Tsai commented on HBASE-19266: Will commit it later without the ugly hack. > TestAcidGuarantees should cover adaptive in-memory compaction > - > > Key: HBASE-19266 > URL: https://issues.apache.org/jira/browse/HBASE-19266 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Chia-Ping Tsai >Priority: Minor > Attachments: HBASE-19266.v0.patch, HBASE-19266.v1.patch > > > Currently TestAcidGuarantees populates 3 policies of (in-memory) compaction. > Adaptive in-memory compaction is new and should be added as 4th compaction > policy. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19266) TestAcidGuarantees should cover adaptive in-memory compaction
[
https://issues.apache.org/jira/browse/HBASE-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264436#comment-16264436
]
Hadoop QA commented on HBASE-19266:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
53s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
48s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
50m 57s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 87m
39s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}165m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19266 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12899055/HBASE-19266.v1.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 8ef4d1bb6d4c 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3e2941a49e |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/9991/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/9991/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> TestAcidGuarantees should cover adaptive in-memory compaction
> -
>
> Key: HBASE-19266
>
[jira] [Commented] (HBASE-19266) TestAcidGuarantees should cover adaptive in-memory compaction
[ https://issues.apache.org/jira/browse/HBASE-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264413#comment-16264413 ] Chia-Ping Tsai commented on HBASE-19266: sure...:P > TestAcidGuarantees should cover adaptive in-memory compaction > - > > Key: HBASE-19266 > URL: https://issues.apache.org/jira/browse/HBASE-19266 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Chia-Ping Tsai >Priority: Minor > Attachments: HBASE-19266.v0.patch, HBASE-19266.v1.patch > > > Currently TestAcidGuarantees populates 3 policies of (in-memory) compaction. > Adaptive in-memory compaction is new and should be added as 4th compaction > policy. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-19266) TestAcidGuarantees should cover adaptive in-memory compaction
[ https://issues.apache.org/jira/browse/HBASE-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264389#comment-16264389 ] Ted Yu edited comment on HBASE-19266 at 11/23/17 2:39 PM: -- I searched the test output on my disk where I don't see the 'ABORTING region server' log. This was probably due to my running eager compaction policy alone and with SSD, the negative memstore size never accumulated during the test run (I observed fluctuating memstore size across zero). It seems separating the compaction policies into their own test would avert the aborting case. In production, it is unlikely that different policies would be used during the lifetime of the region server process. was (Author: yuzhih...@gmail.com): I searched the test output on my disk where I don't see the 'ABORTING region server' log. This was probably due to my running eager compaction policy alone. It seems separating the compaction policies into their own test would avert the aborting case. In production, it is unlikely that different policies would be used during the lifetime of the region server process. > TestAcidGuarantees should cover adaptive in-memory compaction > - > > Key: HBASE-19266 > URL: https://issues.apache.org/jira/browse/HBASE-19266 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Chia-Ping Tsai >Priority: Minor > Attachments: HBASE-19266.v0.patch, HBASE-19266.v1.patch > > > Currently TestAcidGuarantees populates 3 policies of (in-memory) compaction. > Adaptive in-memory compaction is new and should be added as 4th compaction > policy. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19266) TestAcidGuarantees should cover adaptive in-memory compaction
[
https://issues.apache.org/jira/browse/HBASE-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264409#comment-16264409
]
Ted Yu commented on HBASE-19266:
lgtm
Drop the following print upon commit.
{code}
+System.out.println("[CHIA] delete start");
UTIL.deleteTable(TABLE_NAME);
+System.out.println("[CHIA] delete end");
{code}
> TestAcidGuarantees should cover adaptive in-memory compaction
> -
>
> Key: HBASE-19266
> URL: https://issues.apache.org/jira/browse/HBASE-19266
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Attachments: HBASE-19266.v0.patch, HBASE-19266.v1.patch
>
>
> Currently TestAcidGuarantees populates 3 policies of (in-memory) compaction.
> Adaptive in-memory compaction is new and should be added as 4th compaction
> policy.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19266) TestAcidGuarantees should cover adaptive in-memory compaction
[ https://issues.apache.org/jira/browse/HBASE-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264389#comment-16264389 ] Ted Yu commented on HBASE-19266: I searched the test output on my disk where I don't see the 'ABORTING region server' log. This was probably due to my running eager compaction policy alone. It seems separating the compaction policies into their own test would avert the aborting case. In production, it is unlikely that different policies would be used during the lifetime of the region server process. > TestAcidGuarantees should cover adaptive in-memory compaction > - > > Key: HBASE-19266 > URL: https://issues.apache.org/jira/browse/HBASE-19266 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Chia-Ping Tsai >Priority: Minor > Attachments: HBASE-19266.v0.patch, HBASE-19266.v1.patch > > > Currently TestAcidGuarantees populates 3 policies of (in-memory) compaction. > Adaptive in-memory compaction is new and should be added as 4th compaction > policy. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19335) Fix waitUntilAllRegionsAssigned
[
https://issues.apache.org/jira/browse/HBASE-19335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264372#comment-16264372
]
Chia-Ping Tsai commented on HBASE-19335:
Ran {{TestRegionObserverInterface}} with patch 50 times. All passes.
> Fix waitUntilAllRegionsAssigned
> ---
>
> Key: HBASE-19335
> URL: https://issues.apache.org/jira/browse/HBASE-19335
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-19335.master.001.patch
>
>
> Found when debugging flaky test TestRegionObserverInterface#testRecovery.
> In the end, the test does the following:
> - Kills the RS
> - Waits for all regions to be assigned
> - Some validation (unrelated)
> - Cleanup: delete table.
> {noformat}
> cluster.killRegionServer(rs1.getRegionServer().getServerName());
> Threads.sleep(1000); // Let the kill soak in.
> util.waitUntilAllRegionsAssigned(tableName);
> LOG.info("All regions assigned");
> verifyMethodResult(SimpleRegionObserver.class,
> new String[] { "getCtPreReplayWALs", "getCtPostReplayWALs",
> "getCtPreWALRestore",
> "getCtPostWALRestore", "getCtPrePut", "getCtPostPut" },
> tableName, new Integer[] { 1, 1, 2, 2, 0, 0 });
> } finally {
> util.deleteTable(tableName);
> table.close();
> }
> }
> {noformat}
> However, looking at test logs, found that we had overlapping Assigns with
> Unassigns. As a result, regions ended up 'stuck in RIT' and the test timeout.
> Assigns were from the ServerCrashRecovery and Unassigns were from the
> deleteTable cleanup.
> Which begs the question, why did HBTU.waitUntilAllRegionsAssigned(tableName)
> not wait until recovery was complete.
> Answer: Looks like that function is only meant for sunny scenarios but not
> for crashes. It iterates over meta and just [checks for *some value* in the
> server
> column|https://github.com/apache/hbase/blob/cdc2bb17ff38dcbd273cf501aea565006e995a06/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java#L3421]
> which is obviously present and equal to the server that was just killed.
> This bug must be affecting other fault tolerance tests too and fixing it may
> fix more than just one test, hopefully.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-19335) Fix waitUntilAllRegionsAssigned
[
https://issues.apache.org/jira/browse/HBASE-19335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264372#comment-16264372
]
Chia-Ping Tsai edited comment on HBASE-19335 at 11/23/17 2:03 PM:
--
Ran {{TestRegionObserverInterface}} with patch 50 times. All pass.
was (Author: chia7712):
Ran {{TestRegionObserverInterface}} with patch 50 times. All passes.
> Fix waitUntilAllRegionsAssigned
> ---
>
> Key: HBASE-19335
> URL: https://issues.apache.org/jira/browse/HBASE-19335
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-19335.master.001.patch
>
>
> Found when debugging flaky test TestRegionObserverInterface#testRecovery.
> In the end, the test does the following:
> - Kills the RS
> - Waits for all regions to be assigned
> - Some validation (unrelated)
> - Cleanup: delete table.
> {noformat}
> cluster.killRegionServer(rs1.getRegionServer().getServerName());
> Threads.sleep(1000); // Let the kill soak in.
> util.waitUntilAllRegionsAssigned(tableName);
> LOG.info("All regions assigned");
> verifyMethodResult(SimpleRegionObserver.class,
> new String[] { "getCtPreReplayWALs", "getCtPostReplayWALs",
> "getCtPreWALRestore",
> "getCtPostWALRestore", "getCtPrePut", "getCtPostPut" },
> tableName, new Integer[] { 1, 1, 2, 2, 0, 0 });
> } finally {
> util.deleteTable(tableName);
> table.close();
> }
> }
> {noformat}
> However, looking at test logs, found that we had overlapping Assigns with
> Unassigns. As a result, regions ended up 'stuck in RIT' and the test timeout.
> Assigns were from the ServerCrashRecovery and Unassigns were from the
> deleteTable cleanup.
> Which begs the question, why did HBTU.waitUntilAllRegionsAssigned(tableName)
> not wait until recovery was complete.
> Answer: Looks like that function is only meant for sunny scenarios but not
> for crashes. It iterates over meta and just [checks for *some value* in the
> server
> column|https://github.com/apache/hbase/blob/cdc2bb17ff38dcbd273cf501aea565006e995a06/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java#L3421]
> which is obviously present and equal to the server that was just killed.
> This bug must be affecting other fault tolerance tests too and fixing it may
> fix more than just one test, hopefully.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264353#comment-16264353
]
Ted Yu commented on HBASE-19338:
+1
Nice finding.
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: HBASE-19338.master.001.patch,
> HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19242) Add MOB compact support for AsyncAdmin
[
https://issues.apache.org/jira/browse/HBASE-19242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Balazs Meszaros updated HBASE-19242:
Attachment: HBASE-19242.master.001.patch
> Add MOB compact support for AsyncAdmin
> --
>
> Key: HBASE-19242
> URL: https://issues.apache.org/jira/browse/HBASE-19242
> Project: HBase
> Issue Type: Sub-task
> Components: Admin, mob
>Reporter: Duo Zhang
>Assignee: Balazs Meszaros
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19242.master.001.patch
>
>
> {code}
> private CompletableFuture compact(TableName tableName, byte[]
> columnFamily, boolean major,
> CompactType compactType) {
> if (CompactType.MOB.equals(compactType)) {
> // TODO support MOB compact.
> return failedFuture(new UnsupportedOperationException("MOB compact does
> not support"));
> }
> {code}
> We need to support it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19242) Add MOB compact support for AsyncAdmin
[
https://issues.apache.org/jira/browse/HBASE-19242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Balazs Meszaros updated HBASE-19242:
Status: Patch Available (was: In Progress)
> Add MOB compact support for AsyncAdmin
> --
>
> Key: HBASE-19242
> URL: https://issues.apache.org/jira/browse/HBASE-19242
> Project: HBase
> Issue Type: Sub-task
> Components: Admin, mob
>Reporter: Duo Zhang
>Assignee: Balazs Meszaros
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19242.master.001.patch
>
>
> {code}
> private CompletableFuture compact(TableName tableName, byte[]
> columnFamily, boolean major,
> CompactType compactType) {
> if (CompactType.MOB.equals(compactType)) {
> // TODO support MOB compact.
> return failedFuture(new UnsupportedOperationException("MOB compact does
> not support"));
> }
> {code}
> We need to support it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
binlijin updated HBASE-19338:
-
Status: Patch Available (was: Open)
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: HBASE-19338.master.001.patch,
> HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
binlijin updated HBASE-19338:
-
Attachment: HBASE-19338.master.002.patch
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: HBASE-19338.master.001.patch,
> HBASE-19338.master.002.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264290#comment-16264290
]
binlijin commented on HBASE-19338:
--
yscb put qps 8w(without patch) -> 9.8W(with patch)
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: HBASE-19338.master.001.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
[
https://issues.apache.org/jira/browse/HBASE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
binlijin updated HBASE-19338:
-
Attachment: HBASE-19338.master.001.patch
> Performance regression in RegionServerRpcQuotaManager to get ugi
> -
>
> Key: HBASE-19338
> URL: https://issues.apache.org/jira/browse/HBASE-19338
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0-beta-2
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: HBASE-19338.master.001.patch
>
>
> we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
> and have some finding.
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
> prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
> [0x7fc50fafa000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
> - waiting to lock <0x7fcaedc20830> (a java.lang.Class for
> org.apache.hadoop.security.UserGroupInformation)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
> at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
> at
> org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-19338) Performance regression in RegionServerRpcQuotaManager to get ugi
binlijin created HBASE-19338:
Summary: Performance regression in RegionServerRpcQuotaManager to
get ugi
Key: HBASE-19338
URL: https://issues.apache.org/jira/browse/HBASE-19338
Project: HBase
Issue Type: Improvement
Affects Versions: 3.0.0, 2.0.0-beta-2
Reporter: binlijin
Assignee: binlijin
Priority: Critical
we find hbase-2.0.0-beta-1.SNAPSHOT have performance regression with yscb put
and have some finding.
{code}
"RpcServer.default.FPBQ.Fifo.handler=131,queue=17,port=16020" #245 daemon
prio=5 os_prio=0 tid=0x7fc82b22e000 nid=0x3a5db waiting for monitor entry
[0x7fc50fafa000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
- waiting to lock <0x7fcaedc20830> (a java.lang.Class for
org.apache.hadoop.security.UserGroupInformation)
at
org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:264)
at org.apache.hadoop.hbase.security.User.getCurrent(User.java:162)
at
org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:179)
at
org.apache.hadoop.hbase.quotas.RegionServerRpcQuotaManager.checkQuota(RegionServerRpcQuotaManager.java:162)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2521)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:325)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:305)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19266) TestAcidGuarantees should cover adaptive in-memory compaction
[ https://issues.apache.org/jira/browse/HBASE-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19266: --- Status: Open (was: Patch Available) > TestAcidGuarantees should cover adaptive in-memory compaction > - > > Key: HBASE-19266 > URL: https://issues.apache.org/jira/browse/HBASE-19266 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Chia-Ping Tsai >Priority: Minor > Attachments: HBASE-19266.v0.patch, HBASE-19266.v1.patch > > > Currently TestAcidGuarantees populates 3 policies of (in-memory) compaction. > Adaptive in-memory compaction is new and should be added as 4th compaction > policy. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19266) TestAcidGuarantees should cover adaptive in-memory compaction
[ https://issues.apache.org/jira/browse/HBASE-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19266: --- Attachment: HBASE-19266.v1.patch > TestAcidGuarantees should cover adaptive in-memory compaction > - > > Key: HBASE-19266 > URL: https://issues.apache.org/jira/browse/HBASE-19266 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Chia-Ping Tsai >Priority: Minor > Attachments: HBASE-19266.v0.patch, HBASE-19266.v1.patch > > > Currently TestAcidGuarantees populates 3 policies of (in-memory) compaction. > Adaptive in-memory compaction is new and should be added as 4th compaction > policy. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19266) TestAcidGuarantees should cover adaptive in-memory compaction
[ https://issues.apache.org/jira/browse/HBASE-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19266: --- Status: Patch Available (was: Open) > TestAcidGuarantees should cover adaptive in-memory compaction > - > > Key: HBASE-19266 > URL: https://issues.apache.org/jira/browse/HBASE-19266 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Chia-Ping Tsai >Priority: Minor > Attachments: HBASE-19266.v0.patch, HBASE-19266.v1.patch > > > Currently TestAcidGuarantees populates 3 policies of (in-memory) compaction. > Adaptive in-memory compaction is new and should be added as 4th compaction > policy. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-16868) Add a replicate_all flag to avoid misuse the namespaces and table-cfs config of replication peer
[
https://issues.apache.org/jira/browse/HBASE-16868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264241#comment-16264241
]
Hudson commented on HBASE-16868:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4103 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/4103/])
HBASE-16868 Add a replicate_all flag to avoid misuse the namespaces and (zghao:
rev 3e2941a49e58080618fd2d2e6757694c96651e0a)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/master/TestTableCFsUpdater.java
* (edit) hbase-shell/src/main/ruby/shell.rb
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationPeerConfigUtil.java
* (edit)
hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeersZKImpl.java
* (edit) hbase-shell/src/main/ruby/hbase/replication_admin.rb
* (add) hbase-shell/src/main/ruby/shell/commands/set_peer_replicate_all.rb
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeerConfig.java
* (add)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationPeerConfigUpgrader.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestPerTableCFReplication.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdminWithClusters.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncReplicationAdminApi.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdmin.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationWALEntryFilters.java
* (edit) hbase-shell/src/main/ruby/shell/commands/list_peers.rb
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit) hbase-shell/src/test/ruby/hbase/replication_admin_test.rb
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestNamespaceReplication.java
* (edit) hbase-protocol-shaded/src/main/protobuf/Replication.proto
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/NamespaceTableCfWALEntryFilter.java
* (delete)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/TableCFsUpdater.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/ReplicationManager.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncReplicationAdminApiWithClusters.java
> Add a replicate_all flag to avoid misuse the namespaces and table-cfs config
> of replication peer
>
>
> Key: HBASE-16868
> URL: https://issues.apache.org/jira/browse/HBASE-16868
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-16868.master.001.patch,
> HBASE-16868.master.002.patch, HBASE-16868.master.003.patch,
> HBASE-16868.master.004.patch, HBASE-16868.master.005.patch,
> HBASE-16868.master.006.patch, HBASE-16868.master.007.patch,
> HBASE-16868.master.008.patch, HBASE-16868.master.009.patch,
> HBASE-16868.master.010.patch, HBASE-16868.master.011.patch
>
>
> First add a new peer by shell cmd.
> {code}
> add_peer '1', CLUSTER_KEY => "server1.cie.com:2181:/hbase".
> {code}
> If we don't set namespaces and table cfs in peer config. It means replicate
> all tables to the peer cluster.
> Then append a table to the peer config.
> {code}
> append_peer_tableCFs '1', {"table1" => []}
> {code}
> Then this peer will only replicate table1 to the peer cluster. It changes to
> replicate only one table from replicate all tables in the cluster. It is very
> easy to misuse in production cluster. So we should avoid appending table to a
> peer which replicates all table.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19092) Make Tag IA.LimitedPrivate and expose for CPs
[
https://issues.apache.org/jira/browse/HBASE-19092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264232#comment-16264232
]
Chia-Ping Tsai commented on HBASE-19092:
Some trivial comment are in rb. LGTM overall. HBASE-19112 would complete the
{{RawCell}} so +1
> Make Tag IA.LimitedPrivate and expose for CPs
> -
>
> Key: HBASE-19092
> URL: https://issues.apache.org/jira/browse/HBASE-19092
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19092-branch-2.patch,
> HBASE-19092-branch-2_5.patch, HBASE-19092-branch-2_5.patch,
> HBASE-19092.branch-2.0.02.patch, HBASE-19092_001-branch-2.patch,
> HBASE-19092_001.patch, HBASE-19092_002-branch-2.patch, HBASE-19092_002.patch,
> HBASE-19092_004.patch, HBASE-19092_3.patch, HBASE-19092_4.patch
>
>
> We need to make tags as LimitedPrivate as some use cases are trying to use
> tags like timeline server. The same topic was discussed in dev@ and also in
> HBASE-18995.
> Shall we target this for beta1 - cc [~saint@gmail.com].
> So once we do this all related Util methods and APIs should also move to
> LimitedPrivate Util classes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18946) Stochastic load balancer assigns replica regions to the same RS
[ https://issues.apache.org/jira/browse/HBASE-18946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264225#comment-16264225 ] ramkrishna.s.vasudevan commented on HBASE-18946: bq.Man. We want round-robin but with caveats.. "Round-robin except..." I think you are worried about this part. But I see this way. We are still doing round robin. 1) Assign primary replicas (round robin) 2) assign sec replicas ( again round robin just avoid primary replica servers). 3) assign tertiary replicas (again round robin but avoid servers in first two steps). and so on. .. So you think why not do a round robin with the entire set of regions. That is true round robin. But for that some where we need to hold up the regions and bulk it up before going for Assign procs. Otherwise another thing we can do is that, Add a API in AM that will do tentative round robin with the available servers and regions. And in the CreateTableProc before creating the AssignProcs add the target server also to it. So that we go ahead and assign accordingly only. bq.Could we pass the AM the new table regions and ask it to return us plans to use assigning? Reading this comment of yours -you suggest the same I believe? Can we do that only for tables with replicas? > Stochastic load balancer assigns replica regions to the same RS > --- > > Key: HBASE-18946 > URL: https://issues.apache.org/jira/browse/HBASE-18946 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-alpha-3 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-18946.patch, HBASE-18946.patch, > HBASE-18946_2.patch, HBASE-18946_2.patch, > TestRegionReplicasWithRestartScenarios.java > > > Trying out region replica and its assignment I can see that some times the > default LB Stocahstic load balancer assigns replica regions to the same RS. > This happens when we have 3 RS checked in and we have a table with 3 > replicas. When a RS goes down then the replicas being assigned to same RS is > acceptable but the case when we have enough RS to assign this behaviour is > undesirable and does not solve the purpose of replicas. > [~huaxiang] and [~enis]. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18946) Stochastic load balancer assigns replica regions to the same RS
[ https://issues.apache.org/jira/browse/HBASE-18946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-18946: --- Status: Open (was: Patch Available) > Stochastic load balancer assigns replica regions to the same RS > --- > > Key: HBASE-18946 > URL: https://issues.apache.org/jira/browse/HBASE-18946 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-alpha-3 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-18946.patch, HBASE-18946.patch, > HBASE-18946_2.patch, HBASE-18946_2.patch, > TestRegionReplicasWithRestartScenarios.java > > > Trying out region replica and its assignment I can see that some times the > default LB Stocahstic load balancer assigns replica regions to the same RS. > This happens when we have 3 RS checked in and we have a table with 3 > replicas. When a RS goes down then the replicas being assigned to same RS is > acceptable but the case when we have enough RS to assign this behaviour is > undesirable and does not solve the purpose of replicas. > [~huaxiang] and [~enis]. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19334) User.runAsLoginUser not work in AccessController because it use a short circuited connection
[
https://issues.apache.org/jira/browse/HBASE-19334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264213#comment-16264213
]
Hadoop QA commented on HBASE-19334:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
53s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
5s{color} | {color:red} hbase-server: The patch generated 2 new + 63 unchanged
- 10 fixed = 65 total (was 73) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
53s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
53m 45s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 87m
28s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}160m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19334 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12899023/HBASE-19334.master.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 594ef73706c3 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3e2941a49e |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/9989/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/9989/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/9989/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically
[jira] [Commented] (HBASE-16868) Add a replicate_all flag to avoid misuse the namespaces and table-cfs config of replication peer
[
https://issues.apache.org/jira/browse/HBASE-16868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264210#comment-16264210
]
Hudson commented on HBASE-16868:
FAILURE: Integrated in Jenkins build HBase-2.0 #901 (See
[https://builds.apache.org/job/HBase-2.0/901/])
HBASE-16868 Add a replicate_all flag to avoid misuse the namespaces and (zghao:
rev ca6e7e68f4468f0c4c8e3abf1922e5ed07262cae)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeerConfig.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationPeerConfigUtil.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestPerTableCFReplication.java
* (edit) hbase-shell/src/test/ruby/hbase/replication_admin_test.rb
* (delete)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/TableCFsUpdater.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdmin.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/NamespaceTableCfWALEntryFilter.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationWALEntryFilters.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdminWithClusters.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit) hbase-protocol-shaded/src/main/protobuf/Replication.proto
* (edit) hbase-shell/src/main/ruby/shell/commands/list_peers.rb
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncReplicationAdminApi.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/master/TestTableCFsUpdater.java
* (edit)
hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeersZKImpl.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/ReplicationManager.java
* (edit) hbase-shell/src/main/ruby/shell.rb
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncReplicationAdminApiWithClusters.java
* (edit) hbase-shell/src/main/ruby/hbase/replication_admin.rb
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestNamespaceReplication.java
* (add) hbase-shell/src/main/ruby/shell/commands/set_peer_replicate_all.rb
* (add)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationPeerConfigUpgrader.java
> Add a replicate_all flag to avoid misuse the namespaces and table-cfs config
> of replication peer
>
>
> Key: HBASE-16868
> URL: https://issues.apache.org/jira/browse/HBASE-16868
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-16868.master.001.patch,
> HBASE-16868.master.002.patch, HBASE-16868.master.003.patch,
> HBASE-16868.master.004.patch, HBASE-16868.master.005.patch,
> HBASE-16868.master.006.patch, HBASE-16868.master.007.patch,
> HBASE-16868.master.008.patch, HBASE-16868.master.009.patch,
> HBASE-16868.master.010.patch, HBASE-16868.master.011.patch
>
>
> First add a new peer by shell cmd.
> {code}
> add_peer '1', CLUSTER_KEY => "server1.cie.com:2181:/hbase".
> {code}
> If we don't set namespaces and table cfs in peer config. It means replicate
> all tables to the peer cluster.
> Then append a table to the peer config.
> {code}
> append_peer_tableCFs '1', {"table1" => []}
> {code}
> Then this peer will only replicate table1 to the peer cluster. It changes to
> replicate only one table from replicate all tables in the cluster. It is very
> easy to misuse in production cluster. So we should avoid appending table to a
> peer which replicates all table.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19266) TestAcidGuarantees should cover adaptive in-memory compaction
[
https://issues.apache.org/jira/browse/HBASE-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264204#comment-16264204
]
Chia-Ping Tsai commented on HBASE-19266:
bq. The above occurred multiple times during a run with EAGER policy. It would
not cause the region server to shutdown.
The error I mentioned is shown below.
{quote}
2017-11-22 17:24:53,493 FATAL [RS_CLOSE_REGION-asf911:51862-0]
regionserver.HRegionServer(2376): * ABORTING region server
asf911.gq1.ygridcore.net,51862,1511371099686: Assertion failed while closing
store TestAcidGuarantees,,1511371469395.68d2d57885546cb47faa7361cbbbcca7. A.
flushableSize expected=0, actual= dataSize=2290 , heapSize=7008. Current
memstoreSize=-34350. Maybe a coprocessor operation failed and left the memstore
in a partially updated state. *
2017-11-22 17:24:53,493 FATAL [RS_CLOSE_REGION-asf911:51862-0]
regionserver.HRegionServer(2382): RegionServer abort: loaded coprocessors are:
[org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint]
2017-11-22 17:24:53,557 INFO [RS_CLOSE_REGION-asf911:51862-0]
regionserver.HRegionServer(2386): Dump of metrics as JSON on abort: {
{quote}
The detail log is
[here|https://builds.apache.org/job/HBASE-Flaky-Tests/23491/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestAcidGuarantees-output.txt/*view*/].
The code about aborting the rs is shown below.
{code:title=HRegion.class}
for (HStore store : stores.values()) {
MemStoreSize flushableSize = store.getFlushableSize();
if (!(abort || flushableSize.getDataSize() == 0 ||
writestate.readOnly)) {
if (getRegionServerServices() != null) {
getRegionServerServices().abort("Assertion failed while closing
store "
+ getRegionInfo().getRegionNameAsString() + " " + store
+ ". flushableSize expected=0, actual= " + flushableSize
+ ". Current memstoreSize=" + getMemStoreSize() + ". Maybe a
coprocessor "
+ "operation failed and left the memstore in a partially
updated state.", null);
}
}
{code}
bq. TestAcidGuaranteesWithBasicPoli didn't finish in the QA run.
I have ran TestAcidGuaranteesWithBasicPolicy 50 times, and I don't encounter
the hang...Let me try the v2 patch (asap)
{quote}
w.r.t. 'memstoreSize to a negative value' error, the first occurrence is in
TestAcidGuarantees#testMixedAtomicity
However, if I run the subtest alone, it passes with EAGER policy.
{quote}
It seems the error happens by chance. I will dig in after committing this issue.
> TestAcidGuarantees should cover adaptive in-memory compaction
> -
>
> Key: HBASE-19266
> URL: https://issues.apache.org/jira/browse/HBASE-19266
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Attachments: HBASE-19266.v0.patch
>
>
> Currently TestAcidGuarantees populates 3 policies of (in-memory) compaction.
> Adaptive in-memory compaction is new and should be added as 4th compaction
> policy.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18988) Add release managers to reference guide
[ https://issues.apache.org/jira/browse/HBASE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi updated HBASE-18988: -- Fix Version/s: 3.0.0 > Add release managers to reference guide > --- > > Key: HBASE-18988 > URL: https://issues.apache.org/jira/browse/HBASE-18988 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Trivial > Labels: beginner > Fix For: 3.0.0 > > Attachments: HBASE-18988.master.001.patch > > > Reference guide lists release managers only up to version 1.3. We should have > a complete list there. > http://hbase.apache.org/book.html#_release_managers -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19266) TestAcidGuarantees should cover adaptive in-memory compaction
[ https://issues.apache.org/jira/browse/HBASE-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264198#comment-16264198 ] Ted Yu commented on HBASE-19266: Stepping back to the commit which added EAGER to TestAcidGuarantees, HBASE-17575: Run critical tests with each of the Inmemory Compaction Policies enabled 'memstoreSize to a negative value' error can be observed in test output. > TestAcidGuarantees should cover adaptive in-memory compaction > - > > Key: HBASE-19266 > URL: https://issues.apache.org/jira/browse/HBASE-19266 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Chia-Ping Tsai >Priority: Minor > Attachments: HBASE-19266.v0.patch > > > Currently TestAcidGuarantees populates 3 policies of (in-memory) compaction. > Adaptive in-memory compaction is new and should be added as 4th compaction > policy. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-19301: --- Attachment: HBASE-19301-addendum.patch > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301-addendum.patch, HBASE-19301.patch, > HBASE-19301_V2.patch, HBASE-19301_V2.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19337) AsyncMetaTableAccessor may hang when call ScanController.terminate many times
[
https://issues.apache.org/jira/browse/HBASE-19337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264172#comment-16264172
]
Hadoop QA commented on HBASE-19337:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
39s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
6s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
57m 54s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
56s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19337 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12899031/HBASE-19337.master.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 6e7b69af41ee 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 3e2941a49e |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/9990/testReport/ |
| modules | C: hbase-client U: hbase-client |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/9990/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> AsyncMetaTableAccessor may hang when call
[jira] [Commented] (HBASE-18298) RegionServerServices Interface cleanup for CP expose
[ https://issues.apache.org/jira/browse/HBASE-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264132#comment-16264132 ] Ankit Singhal commented on HBASE-18298: --- Thank you [~anoop.hbase] and @Stack for taking a look and suggestions. bq. You need to abort because you fail to offline an index table? Tell us more. Actually, when we are unable to write to an Index table , we try to disable the index by updating the same in our meta table(SYSTEM.CATALOG) so that we will not attempt any future writes but if we fail to update meta table, to avoid inconsistency and expecting that writes will not succeed again, we kill the server so that the edits in the WALs can be replayed on restart. bq. IIRC, you can throw an exception and we will abort. We can add a test for this. yes, we can utilize this. bq. is config of abort on Exception is to be set false for Phoenix usage? In such case RS wont get aborted. Only that particular CP will get unloaded No, we need it to be true. Either we should ensure at Phoenix side that this config is never set to false on the server or we can have a special exception from HBase for forceful abort. Raised PHOENIX-4399 for the same. > RegionServerServices Interface cleanup for CP expose > > > Key: HBASE-18298 > URL: https://issues.apache.org/jira/browse/HBASE-18298 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18298.patch, HBASE-18298_V2.patch, > HBASE-18298_V3.patch, HBASE-18298_V4.patch, HBASE-18298_V5.patch, > HBASE-18298_V6.patch, HBASE-18298_V7.patch, HBASE-18298_V7.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19266) TestAcidGuarantees should cover adaptive in-memory compaction
[
https://issues.apache.org/jira/browse/HBASE-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264124#comment-16264124
]
Ted Yu commented on HBASE-19266:
Note: the error is only a log:
{code}
LOG.error("Asked to modify this region's (" + this.toString()
+ ") memstoreSize to a negative value which is incorrect. Current
memstoreSize="
+ (memstoreDataSize - delta) + ", delta=" + delta, new Exception());
{code}
The above occurred multiple times during a run with EAGER policy. It would not
cause the region server to shutdown.
> TestAcidGuarantees should cover adaptive in-memory compaction
> -
>
> Key: HBASE-19266
> URL: https://issues.apache.org/jira/browse/HBASE-19266
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Attachments: HBASE-19266.v0.patch
>
>
> Currently TestAcidGuarantees populates 3 policies of (in-memory) compaction.
> Adaptive in-memory compaction is new and should be added as 4th compaction
> policy.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
