Hi Ted:
Following is part of log. I have changed log level to TRACE. But still can’t
find any clue.
2016-07-22 12:03:38,261 DEBUG [ProcedureExecutor-1]
procedure2.ProcedureExecutor: Procedure completed in 1.6310sec:
DisableTableProcedure (table=testTbl) id=109 owner=shengchen.ma state=F
INISHED
2016-07-22 12:03:38,866 TRACE
[RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479] ipc.RpcServer:
RequestHeader call_id: 14 method_name: "IsMasterRunning" request_param: true
totalRequestSize: 23 bytes
2016-07-22 12:03:38,866 TRACE
[B.defaultRpcServer.handler=24,queue=0,port=39479] ipc.RpcServer: callId: 14
service: MasterService methodName: IsMasterRunning size: 23 connection:
192.168.3.10:49752 executing as shengchen.ma
2016-07-22 12:03:38,866 TRACE
[B.defaultRpcServer.handler=24,queue=0,port=39479] ipc.RpcServer: callId: 14
service: MasterService methodName: IsMasterRunning size: 23 connection:
192.168.3.10:49752 param: TODO: class
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$IsMasterRunningRequest
connection: 192.168.3.10:49752, response is_master_running: true queueTime: 0
processingTime: 0 totalTime: 0
2016-07-22 12:03:38,878 TRACE
[RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479] ipc.RpcServer:
RequestHeader call_id: 15 method_name: "getProcedureResult" request_param: true
totalRequestSize: 28 bytes
2016-07-22 12:03:38,878 TRACE [B.defaultRpcServer.handler=7,queue=1,port=39479]
ipc.RpcServer: callId: 15 service: MasterService methodName: getProcedureResult
size: 28 connection: 192.168.3.10:49752 executing as shengchen.ma
2016-07-22 12:03:38,878 DEBUG [B.defaultRpcServer.handler=7,queue=1,port=39479]
master.MasterRpcServices: Checking to see if procedure is done procId=109
2016-07-22 12:03:38,878 TRACE [B.defaultRpcServer.handler=7,queue=1,port=39479]
ipc.RpcServer: callId: 15 service: MasterService methodName: getProcedureResult
size: 28 connection: 192.168.3.10:49752 param: TODO: class
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$GetProcedureResultRequest
connection: 192.168.3.10:49752, response state: FINISHED start_time:
1469189016526 last_update: 1469189018157 queueTime: 0 processingTime: 0
totalTime: 0
2016-07-22 12:03:38,888 TRACE
[RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479] ipc.RpcServer:
RequestHeader call_id: 16 method_name: "IsMasterRunning" request_param: true
totalRequestSize: 23 bytes
2016-07-22 12:03:38,888 TRACE [B.defaultRpcServer.handler=9,queue=0,port=39479]
ipc.RpcServer: callId: 16 service: MasterService methodName: IsMasterRunning
size: 23 connection: 192.168.3.10:49752 executing as shengchen.ma
2016-07-22 12:03:38,888 TRACE [B.defaultRpcServer.handler=9,queue=0,port=39479]
ipc.RpcServer: callId: 16 service: MasterService methodName: IsMasterRunning
size: 23 connection: 192.168.3.10:49752 param: TODO: class
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$IsMasterRunningRequest
connection: 192.168.3.10:49752, response is_master_running: true queueTime: 0
processingTime: 0 totalTime: 0
2016-07-22 12:03:39,007 TRACE
[RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479] ipc.RpcServer:
RequestHeader call_id: 17 method_name: "ModifyTable" request_param: true
totalRequestSize: 505 bytes
2016-07-22 12:03:39,007 TRACE
[B.defaultRpcServer.handler=17,queue=2,port=39479] ipc.RpcServer: callId: 17
service: MasterService methodName: ModifyTable size: 505 connection:
192.168.3.10:49752 executing as shengchen.ma
2016-07-22 12:03:39,047 DEBUG
[B.defaultRpcServer.handler=17,queue=2,port=39479] util.CoprocessorClassLoader:
Skipping exempt class
org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint -
delegating directly to parent
2016-07-22 12:03:39,048 DEBUG
[B.defaultRpcServer.handler=17,queue=2,port=39479] ipc.RpcServer:
B.defaultRpcServer.handler=17,queue=2,port=39479: callId: 17 service:
MasterService methodName: ModifyTable size: 505 connection: 192.168.3.10:49752
org.apache.hadoop.hbase.DoNotRetryIOException: Class
org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint cannot be
loaded Set hbase.table.sanity.checks to false at conf or table descriptor if
you want to bypass sanity checks
at
org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1680)
at
org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1541)
at org.apache.hadoop.hbase.master.HMaster.modifyTable(HMaster.java:2028)
at
org.apache.hadoop.hbase.master.MasterRpcServices.modifyTable(MasterRpcServices.java:1170)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55680)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)
2016-07-22 12:03:39,622 DEBUG
[RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479] ipc.RpcServer:
RpcServer.listener,port=39479: Caught exception while reading:Connection reset
by peer
2016-07-22 12:03:39,622 DEBUG
[RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479] ipc.RpcServer:
RpcServer.listener,port=39479: DISCONNECTING client 192.168.3.10:49752 because
read count=-1. Number of active connections: 3
2016-07-22 12:03:39,623 WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:17570]
server.NIOServerCnxn: Exception causing close of session 0x1561182c97e0009 due
to java.io.IOException: Connection reset by peer
2016-07-22 12:03:39,623 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:17570]
server.NIOServerCnxn: Closed socket connection for client /192.168.3.10:49751
which had sessionid 0x1561182c97e0009
2016-07-22 12:03:40,127 TRACE
[RpcServer.reader=1,bindAddress=aven01.novalocal,port=39479] ipc.RpcServer:
RequestHeader call_id: 5466 method_name: "RegionServerReport" request_param:
true totalRequestSize: 5748 bytes
2016-07-22 12:03:40,128 TRACE
[B.defaultRpcServer.handler=12,queue=0,port=39479] ipc.RpcServer: callId: 5466
service: RegionServerStatusService methodName: RegionServerReport size: 5.6 K
connection: 192.168.0.17:49483 executing as traf
2016-07-22 12:03:40,128 TRACE
[B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
7e8aa2d93aa716ad2068808d938f0786, existingValue=-1, completeSequenceId=-1
2016-07-22 12:03:40,128 TRACE
[B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
7e8aa2d93aa716ad2068808d938f0786, family=mt_, existingValue=-1,
completeSequenceId=-1
2016-07-22 12:03:40,128 TRACE
[B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
7e8aa2d93aa716ad2068808d938f0786, family=tddlcf, existingValue=-1,
completeSequenceId=-1
2016-07-22 12:03:40,128 TRACE
[B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
3052710e85ab46a686a22e23bff937e2, existingValue=-1, completeSequenceId=-1
2016-07-22 12:03:40,128 TRACE
[B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
3052710e85ab46a686a22e23bff937e2, family=cpf, existingValue=-1,
completeSequenceId=-1
-----邮件原件-----
发件人: Ted Yu [mailto:[email protected]]
发送时间: 2016年7月22日 19:48
收件人: [email protected]
主题: Re: how to Dynamic load of Coprocessors
w.r.t. the DoNotRetryIOException, can you take a look at region server log
where testTbl region(s) was hosted ?
See if there is some clue why the sanity check failed.
Thanks
On Fri, Jul 22, 2016 at 1:12 AM, Ma, Sheng-Chen (Aven) < [email protected]>
wrote:
> Hi all:
> I want to dynamic add coprocessor in order to not restart hbase.
>
> Following is my code:
> Path path = new Path("/coprocessor_jars");
> FileSystem fs = FileSystem.get(conf);
> FileStatus[] status = fs.listStatus(path);
> Path[] listedPaths = FileUtil.stat2Paths(status);
> for (Path p : listedPaths) {
> if(p.getName().contains("test.jar")){
> hdfsPath = p;
> }
> }
> HBaseAdmin hadmin = new HBaseAdmin(conf);
> HTableDescriptor tableDesc =
> hadmin.getTableDescriptor("testTbl".getBytes());
>
> tableDesc.addCoprocessor("org.apache.hadoop.hbase.coprocessor.transact
> ional.TestRegionEndpoint",
> hdfsPath,
> Coprocessor.PRIORITY_USER, null);
> //
> tableDesc.removeCoprocessor("org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint");
> for (Entry<ImmutableBytesWritable, ImmutableBytesWritable> entry :
> tableDesc.getValues().entrySet()) {
> System.out.println(Bytes.toString(entry.getKey().get()) +
> " = " + Bytes.toString(entry.getValue().get()));
> }
> hadmin.disableTable("testTbl".getBytes());
> hadmin.modifyTable("testTbl", tableDesc);
> hadmin.enableTable("testTbl");
>
> the syso print : coprocessor$1 = hdfs://
> 192.168.0.17:17400/coprocessor_jars/test.jar|org.apache.hadoop.hbase.c
> oprocessor.transactional.TestRegionEndpoint|1073741823|
>
> and the remote side return Exception:
> org.apache.hadoop.hbase.DoNotRetryIOException:
> org.apache.hadoop.hbase.DoNotRetryIOException: Class
> org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint
> cannot be loaded Set hbase.table.sanity.checks to false at conf or
> table descriptor if you want to bypass sanity checks
>
> I use hbase 1.2 and the test.jar is not under hbase/lib, I just put
> the test.jar in hdfs.
>
> If I add test.jar to hbase/lib but not restart hbase, the upon code
> still throw same exception.
> If I add test.jar to hbase/lib and restart hbase, the upon code will
> exec successful.
>
> But my requirement is not restart hbase.
>
> Is there someone can give me a favor.
>
> Thanks
>
>
>
