[ 
https://issues.apache.org/jira/browse/YARN-2330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14069943#comment-14069943
 ] 

Zhijie Shen commented on YARN-2330:
-----------------------------------

It is very likely that the history file of the given application is still not 
closed for writing (for example, after RM restarting, RM reopen the history 
file to append the history information). On the other side, the reader want to 
scan the file under writing.

The following logic is broken, because writer is invoked on RM, while reader is 
invoked on timeline server. Hence, from the point of view of reader. 
outstandingWriters is always empty. This cannot be used to indicate whether a 
file was opened for writing or not,
{code}
    // The history file is still under writing
    if (outstandingWriters.containsKey(appId)) {
      throw new IOException("History file for application " + appId
          + " is under writing");
    }
{code}

> Jobs are not displaying in timeline server after RM restart
> -----------------------------------------------------------
>
>                 Key: YARN-2330
>                 URL: https://issues.apache.org/jira/browse/YARN-2330
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: 2.4.1
>         Environment: Nodemanagers 3 (3*8GB)
> Queues A = 70%
> Queues B = 30%
>            Reporter: Nishan Shetty
>
> Submit jobs to queue a
> While job is running Restart RM 
> Observe that those jobs are not displayed in timelineserver
> {code}
> 2014-07-22 10:11:32,084 ERROR 
> org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore:
>  History information of application application_1406002968974_0003 is not 
> included into the result due to the exception
> java.io.IOException: Cannot seek to negative offset
>       at org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1381)
>       at 
> org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:63)
>       at org.apache.hadoop.io.file.tfile.BCFile$Reader.<init>(BCFile.java:624)
>       at org.apache.hadoop.io.file.tfile.TFile$Reader.<init>(TFile.java:804)
>       at 
> org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore$HistoryFileReader.<init>(FileSystemApplicationHistoryStore.java:683)
>       at 
> org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.getHistoryFileReader(FileSystemApplicationHistoryStore.java:661)
>       at 
> org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.getApplication(FileSystemApplicationHistoryStore.java:146)
>       at 
> org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.getAllApplications(FileSystemApplicationHistoryStore.java:199)
>       at 
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getAllApplications(ApplicationHistoryManagerImpl.java:103)
>       at 
> org.apache.hadoop.yarn.server.webapp.AppsBlock.render(AppsBlock.java:75)
>       at 
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:66)
>       at 
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:76)
>       at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
>       at 
> org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49)
>       at 
> org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
>       at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$TD._(Hamlet.java:845)
>       at 
> org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:56)
>       at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:82)
>       at org.apache.hadoop.yarn.webapp.Dispatcher.render(Dispatcher.java:197)
>       at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:156)
>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>       at 
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>       at 
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>       at 
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>       at 
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>       at 
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
>       at 
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
>       at 
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
>       at 
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
>       at 
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>       at 
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>       at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>       at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>       at 
> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
>       at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>       at 
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1192)
>       at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>       at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
>       at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Reply via email to