[
https://issues.apache.org/jira/browse/CHUKWA-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850184#action_12850184
]
Ahmed Fathalla commented on CHUKWA-4:
-------------------------------------
I have replicated this issue on my local development machine. When
stop-collectors.sh runs, the following is logged to the collector log
2010-03-26 14:10:12,988 WARN Shutdown SeqFileWriter - cannot rename dataSink
file:/chukwa/logs/201026140940728_ahmedlaptop_421d1571279a5d08887ffc.chukwa
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:226)
at org.apache.hadoop.hdfs.DFSClient.access$600(DFSClient.java:67)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3219)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3188)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
at
org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.close(SeqFileWriter.java:318)
at
org.apache.hadoop.chukwa.datacollection.writer.SocketTeeWriter.close(SocketTeeWriter.java:267)
at
org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.close(PipelineStageWriter.java:46)
at
org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.destroy(ServletCollector.java:227)
at
org.mortbay.jetty.servlet.ServletHolder.destroyInstance(ServletHolder.java:315)
at
org.mortbay.jetty.servlet.ServletHolder.doStop(ServletHolder.java:286)
at
org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:64)
at
org.mortbay.jetty.servlet.ServletHandler.doStop(ServletHandler.java:170)
at
org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:64)
at
org.mortbay.jetty.handler.HandlerWrapper.doStop(HandlerWrapper.java:142)
at
org.mortbay.jetty.servlet.SessionHandler.doStop(SessionHandler.java:124)
at
org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:64)
at
org.mortbay.jetty.handler.HandlerWrapper.doStop(HandlerWrapper.java:142)
at
org.mortbay.jetty.handler.ContextHandler.doStop(ContextHandler.java:569)
at
org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:64)
at
org.mortbay.jetty.handler.HandlerWrapper.doStop(HandlerWrapper.java:142)
at org.mortbay.jetty.Server.doStop(Server.java:281)
at
org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:64)
at org.mortbay.jetty.Server$ShutdownHookThread.run(Server.java:559)
The line that throws the exception is this.currentOutputStr.close(); in the
close() method of SeqFileWriter.java
try {
lock.acquire();
if
if (this.currentOutputStr != null) {
this.currentOutputStr.close();
}
if(ENABLE_ROTATION_ON_CLOSE)
if(bytesThisRotate > 0)
fs.rename(currentPath, new Path(currentFileName + ".done"));
else
fs.delete(currentPath, false);
} catch (Throwable e) {
log.warn("cannot rename dataSink file:" + currentPath,e);
} finally {
lock.release();
}
}
It seems that the FileSystem is somehow closed before we attempt to close
currentOutputStr. Does anyone have recommendations on how this can be resolved?
> Collectors don't finish writing .done datasink from last .chukwa datasink
> when stopped using bin/stop-collectors
> ----------------------------------------------------------------------------------------------------------------
>
> Key: CHUKWA-4
> URL: https://issues.apache.org/jira/browse/CHUKWA-4
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: data collection
> Environment: I am running on our local cluster. This is a linux
> machine that I also run Hadoop cluster from.
> Reporter: Andy Konwinski
> Priority: Minor
>
> When I use start-collectors, it creates the datasink as expected, writes to
> it as per normal, i.e. writes to the .chukwa file, and roll overs work fine
> when it renames the .chukwa file to .done. However, when I use
> bin/stop-collectors to shut down the running collector it leaves a .chukwa
> file in the HDFS file system. Not sure if this is a valid sink or not, but I
> think that the collector should gracefully clean up the datasink and rename
> it .done before exiting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
