Hi Todd,
you are right I should be more specific:
1. from the namenode log:
2011-12-11 08:57:23,245 WARN
org.apache.hadoop.hdfs.server.common.Storage: rollEdidLog: removing
storage /srv/hadoop/hdfs/edit
2011-12-11 08:57:23,311 WARN
org.apache.hadoop.hdfs.server.common.Storage: incrementCheckpointTime
failed on /srv/hadoop/hdfs/name;type=IMAGE
2011-12-11 08:57:23,316 WARN org.mortbay.log: /getimage:
java.io.IOException: GetImage failed. java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.namenode.GetImageServlet$1.run(GetImageServlet.java:83)
at
org.apache.hadoop.hdfs.server.namenode.GetImageServlet$1.run(GetImageServlet.java:78)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at
org.apache.hadoop.hdfs.server.namenode.GetImageServlet.doGet(GetImageServlet.java:78)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2. Embarrassingly no, we had only, now we have 2 and periodically
backups , :(
3. Yes
4. hadoop version
Hadoop 0.20.2-cdh3u2
Subversion
file:///tmp/nightly_2011-10-13_20-02-02_3/hadoop-0.20-0.20.2+923.142-1~lucid
-r 95a824e4005b2a94fe1c11f1ef9db4c672ba43cb
Compiled by root on Thu Oct 13 21:52:18 PDT 2011
From source with checksum 644e5db6c59d45bca96cec7f220dda51
Thanks,
Guy
On Thu 15 Dec 2011 11:39:26 AM IST, Todd Lipcon wrote:
Hi Guy,
Several questions come to mind here:
- What was the exact WARN level message you saw?
- Did you have multiple dfs.name.dirs configured as recommended by
most setup guides?
- Did you try entering safemode and then running saveNamespace to
persist the image before shutting down the NN? This would have saved
your data.
- What exact version of HDFS were you running?
This is certainly not expected behavior... all of the places where an
edit log fails have a check against there being 0 edit logs remaining
and should issue a FATAL level message followed by a System.exit(-1).
-Todd
On Thu, Dec 15, 2011 at 1:16 AM, Guy Doulberg<[email protected]> wrote:
Hi guys,
We recently had the following problem on our production cluster:
The filesystem containing the editlog and fsimage had no free inodes.
As a result the namenode wasn't able to obtain an inode for the fsimage and
editlog after a checkpiot has been reached, while the previous files were
freed.
Unfortunately, we had no monitoring on the inodes number, so it happens
that the namenode ran in this state for a few hours.
We have noticed this failure in its DFS-status page.
But the namenode didn't enter safe-mode, so all the writes were made
couldn't be persisted to the editlog.
After discovering the problem we freed inodes, and the file-system seemed to
be okay again, we tried to force the namenode to persist to editlog with no
success,
Eventually, we restarted the namenode -which of-course caused us to lose all
the data that was written to the hdfs during these few hours (fortunately we
have backup of the recent writes - so we restored the data from there )
This situation raises some severe concerns,
1. How come the namenode identified a failure in persisting its editlog and
didn't enter safe-mode? (The exception was given only a WARN -severity and
not a CRITICAL)
2. How come after we freed inodes, we couldn't persist the namenode? Maybe
there should be a command in the CLI to should enable us to force the
namenode to persist its editlog
Do you know of a JIRA opened for these issue, or should I open one?
Thanks Guy
