[
https://issues.apache.org/jira/browse/HADOOP-3069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HADOOP-3069:
----------------------------------------
Fix Version/s: (was: 0.17.0)
0.16.3
Status: Patch Available (was: Open)
> A failure on SecondaryNameNode truncates the primary NameNode image.
> --------------------------------------------------------------------
>
> Key: HADOOP-3069
> URL: https://issues.apache.org/jira/browse/HADOOP-3069
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.13.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.16.3
>
> Attachments: TruncatePrimaryImageBug.patch
>
>
> When the primary name-node pulls the new image from the secondary,
> and the transfer fails for some reason then the primary considers the new
> image,
> which may not be completely transfered yet or may be not transfered at all,
> as a valid one and will roll it into the new files system image, which will
> be either corrupted or empty.
> The problem here is that the error message from the secondary node does not
> reach the primary.
> And this happens because TransferFsImage.getFileServer() closes the
> connection output stream
> in its finalize section. The secondary later sends the error reply which
> cannot be received by the primary
> and causes the following exception on the secondary:
> {code}
> 08/03/21 12:16:52 ERROR NameNode.Secondary: java.io.FileNotFoundException:
> \hadoop-data\hdfs\namesecondary\destimage.tmp (The system cannot find the
> file specified)
> 08/03/21 12:16:56 WARN /: /getimage?getimage=1:
> java.lang.IllegalStateException: Committed
> at
> org.mortbay.jetty.servlet.ServletHttpResponse.resetBuffer(ServletHttpResponse.java:212)
> at
> org.mortbay.jetty.servlet.ServletHttpResponse.sendError(ServletHttpResponse.java:375)
> at
> org.apache.hadoop.dfs.SecondaryNameNode$GetImageServlet.doGet(SecondaryNameNode.java:485)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
> at
> org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at
> org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at
> org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
> {code}
> But the exception does not effect the behavior of the primary node. Since the
> stream is closed the primary thinks
> the file transfer was successfully finished and acts further accordingly.
> There 2 bugs that need to be fixed here.
> # The error message should be delivered to the primary, and the primary
> should not corrupt its image in case of an error.
> # The doGet() method of both HttpServlet-s should catch not only
> IOException-s but any exceptions.
> If we miss NPE or SecurityException the main image will truncated.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
