[
https://issues.apache.org/jira/browse/HBASE-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217442#comment-16217442
]
Josh Elser commented on HBASE-19065:
------------------------------------
{noformat}
2017-10-23 23:37:55,199 WARN
[RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=55352]
regionserver.SecureBulkLoadManager(184): unable to add token
org.apache.hadoop.hbase.HBaseIOException: com.google.protobuf.ServiceException:
Error calling method hbase.pb.AuthenticationService.GetAuthenticationToken
at
org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:370)
at
org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:356)
at
org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:74)
at
org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager.secureBulkLoadHFiles(SecureBulkLoadManager.java:178)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.bulkLoadHFile(RSRpcServices.java:2277)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41550)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:134)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258)
Caused by: com.google.protobuf.ServiceException: Error calling method
hbase.pb.AuthenticationService.GetAuthenticationToken
at
org.apache.hadoop.hbase.client.SyncCoprocessorRpcChannel.callBlockingMethod(SyncCoprocessorRpcChannel.java:71)
at
org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService$BlockingStub.getAuthenticationToken(AuthenticationProtos.java:4512)
at
org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:69)
... 7 more
Caused by: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No
registered coprocessor service found for AuthenticationService in region
hbase:meta,,1
at
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7742)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2355)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2337)
at
org.apache.hadoop.hbase.client.RegionCoprocessorRpcChannel$1.rpcCall(RegionCoprocessorRpcChannel.java:87)
at
org.apache.hadoop.hbase.client.RegionCoprocessorRpcChannel$1.rpcCall(RegionCoprocessorRpcChannel.java:81)
at
org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:106)
at
org.apache.hadoop.hbase.client.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:91)
at
org.apache.hadoop.hbase.client.SyncCoprocessorRpcChannel.callBlockingMethod(SyncCoprocessorRpcChannel.java:69)
... 9 more
{noformat}
I looked at the surefire output collected by Yetus on the precommit job for
TestSecureLoadIncrementalHFiles. This was the only thing that jumped out at me
that looked problematic. The rest appear to have been killed by timeout (not
something else).
That said, I don't think that's related to your change here (looks more like a
legitimate test setup issue that has gone unnoticed). +1
> HRegion#bulkLoadHFiles() should wait for concurrent Region#flush() to finish
> ----------------------------------------------------------------------------
>
> Key: HBASE-19065
> URL: https://issues.apache.org/jira/browse/HBASE-19065
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Attachments: 19065.v1.txt, 19065.v2.txt, 19065.v2.txt
>
>
> When I was debugging bulk load failure, I saw the following in region server
> log:
> {code}
> 2017-10-17 23:05:28,795 DEBUG
> [B.defaultRpcServer.handler=0,queue=0,port=16020] regionserver.HRegion: NOT
> flushing memstore for region mx_,
> f449669a8b0341e4edbd2ebdacc72094f449669a8b0341e4edbd2ebdacc7209420150711,1504909319142.52d496ba39036e0c2cc9522895ad438f.,
> flushing=true, writesEnabled=true
> 2017-10-17 23:05:28,796 ERROR
> [B.defaultRpcServer.handler=0,queue=0,port=16020]
> access.SecureBulkLoadEndpoint: Failed to complete bulk load
> java.io.IOException: Could not bulk load with an assigned sequential ID
> because the flush didn't run. Reason for not flushing: Not flushing since
> already flushing
> at
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:5282)
> at
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:292)
> at
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:275)
> {code}
> There was concurrent flush which got misinterpreted by bulkLoadHFiles().
> HRegion#bulkLoadHFiles() should wait for the concurrent flush to complete.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
