Hi guys,
I have an application that gives me error.
Lease renewer daemon for [] with renew id 1 exited
In the superstep 2 of a Giraph Application. For some reason, after
finishing a superstep, the worker don't do much, only gives this as
output:
16/08/30 00:48:35 INFO netty.NettyClient: waitAllRequests: Finished
all requests. MBytes/sec received = 0.0041, MBytesReceived = 0, ave
received req MBytes = 0, secs waited = 0.002
MBytes/sec sent = 0.0079, MBytesSent = 0, ave sent req MBytes = 0,
secs waited = 0.002
16/08/30 00:48:35 DEBUG aggregators.OwnerAggregatorServerData: reset:
Ready for next superstep
16/08/30 00:48:35 DEBUG worker.WorkerAggregatorHandler:
finishSuperstep: Aggregators finished
16/08/30 00:48:56 DEBUG hdfs.LeaseRenewer: Lease renewer daemon for []
with renew id 1 executed
16/08/30 00:48:56 DEBUG hdfs.LeaseRenewer: Lease renewer daemon for []
with renew id 1 expired
16/08/30 00:48:56 DEBUG hdfs.LeaseRenewer: *Lease renewer daemon for
[] with renew id 1 exited*
The only thing i know about this, is that hdfs is creating a Daemon,
but for some reason fails and enter in a finally clause:
synchronized void put(final String src, final DFSOutputStream out,
final DFSClient dfsc) {
if (dfsc.isClientRunning()) {
if (!isRunning() || isRenewerExpired()) {
//start a new deamon with a new id.
final int id = ++currentId;
daemon = new Daemon(new Runnable() {
@Override
public void More ...run() {
try {
if (LOG.isDebugEnabled()) {
LOG.debug("Lease renewer daemon for " + clientsString()
+ " with renew id " + id + " started");
}
LeaseRenewer.this.run(id);
} catch(InterruptedException e) {
if (LOG.isDebugEnabled()) {
LOG.debug(LeaseRenewer.this.getClass().getSimpleName()
+ " is interrupted.", e);
}
} finally {
synchronized(LeaseRenewer.this) {
Factory.INSTANCE.remove(LeaseRenewer.this);
}
if (LOG.isDebugEnabled()) {
* LOG.debug("Lease renewer daemon for " + clientsString()
+ " with renew id " + id + " exited");*
}
}
}
@Override
public String More ...toString() {
return String.valueOf(LeaseRenewer.this);
}
});
daemon.start();
}
dfsc.putFileBeingWritten(src, out);
emptyTime = Long.MAX_VALUE;
}
}
In the same worker, but a little earlier, the worker can create the
HDFS Daemon successfully:
1970,writeDone=true,writeSuccess=true). Waiting on 77
requests*16/08/30 00:48:26 DEBUG hdfs.LeaseRenewer: Lease renewer
daemon for [] with renew id 1 executed*
16/08/30 00:48:26 DEBUG handler.RequestEncoder: write: Client 4,
requestId 534, size = 524891, SEND_WORKER_MESSAGES_REQUEST took
15943119 ns
16/08/30 00:48:26 DEBUG netty.InboundByteCounter: channelRead: [id:
0x74b0ba68, /172.31.45.214:56039 => /172.31.45.213:30004] buffer size
= 262144, total bytes = 720996472
16/08/30 00:48:26 DEBUG netty.OutboundByteCounter: write: [id:
0x9db0a2f6, /172.31.45.213:44192 =>
ip-172-31-45-213.ec2.internal/172.31.45.213:30000] buffer size =
524891, total bytes = 555502090
16/08/30 00:48:26 DEBUG netty.InboundByteCounter: channelRead: [id:
0x74b0ba68, /172.31.45.214:56039 => /172.31.45.213:30004] buffer size
= 262144, total bytes = 721258616
16/08/30 00:48:26 DEBUG netty.InboundByteCounter: channelRead: [id:
0x74b0ba68, /172.31.45.214:56039 => /172.31.45.213:30004] buffer size
= 262144, total bytes = 721520760
16/08/30 00:48:26 DEBUG netty.InboundByteCounter: channelRead: [id:
0x74b0ba68, /172.31.45.214:56039 => /172.31.45.213:30004] buffer size
= 262144, total bytes = 721782904
16/08/30 00:48:26 DEBUG netty.InboundByteCounter: channelRead: [id:
0xb5d6ea70, /172.31.45.213:46826 => /172.31.45.213:30004] buffer size
= 66511, total bytes = 721849415
16/08/30 00:48:26 DEBUG handler.RequestDecoder: decode: Client 0,
requestId 493, SEND_WORKER_MESSAGES_REQUEST, with size 524888 took
581852 ns
16/08/30 00:48:26 DEBUG handler.RequestServerHandler: messageReceived:
Processing client 0, requestId 493, SEND_WORKER_MESSAGES_REQUEST took
31752 ns
16/08/30 00:48:26 DEBUG netty.OutboundByteCounter: write: [id:
0xb5d6ea70, /172.31.45.213:46826 => /172.31.45.213:30004] buffer size
= 13, total bytes = 18876
16/08/30 00:48:26 DEBUG netty.InboundByteCounter: channelRead: [id:
0x9db0a2f6, /172.31.45.213:44192 =>
ip-172-31-45-213.ec2.internal/172.31.45.213:30000] buffer size = 13,
total bytes = 13013
16/08/30 00:48:26 DEBUG handler.ResponseClientHandler:
messageReceived: Completed (taskId =
0)(reqId=459,destAddr=ip-172-31-45-213.ec2.internal:30000,elapsedNanos=1481591427,started=Thu
Jan 01 01:49:56 UTC 1970,writeDone=true,writeSuccess=true). Waiting
on 77 requests
16/08/30 00:48:26 DEBUG netty.InboundByteCounter: channelRead: [id:
0x74b0ba68, /172.31.45.214:56039 => /172.31.45.213:30004] buffer size
= 262144, total bytes = 722111559
16/08/30 00:48:26 DEBUG netty.InboundByteCounter: channelRead: [id:
0x74b0ba68, /172.31.45.214:56039 => /172.31.45.213:30004] buffer size
= 262144, total bytes = 722373703
16/08/30 00:48:26 DEBUG netty.InboundByteCounter: channelRead: [id:
0x74b0ba68, /172.31.45.214:56039 => /172.31.45.213:30004] buffer size
= 262144, total bytes = 722635847
16/08/30 00:48:26 DEBUG netty.InboundByteCounter: channelRead: [id:
0x74b0ba68, /172.31.45.214:56039 => /172.31.45.213:30004] buffer size
= 252147, total bytes = 722887994
Any help will be greatly appreciated!!
The only reference on this error that i could find regards to Spark
user list, and confuses me a lot :D
Bye!
Jose
