[ 
https://issues.apache.org/jira/browse/YARN-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddhi Mehta updated YARN-3578:
-------------------------------
    Summary: Accessing Counters after RM restart results in stale cache (fails 
with FileNotFoundException)  (was: Accessing Counters after RM restart fails 
with FileNotFoundException)

> Accessing Counters after RM restart results in stale cache (fails with 
> FileNotFoundException)
> ---------------------------------------------------------------------------------------------
>
>                 Key: YARN-3578
>                 URL: https://issues.apache.org/jira/browse/YARN-3578
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: yarn
>    Affects Versions: 2.3.0
>            Reporter: Siddhi Mehta
>
> When the job client tries to access counters for a recently completed job.
> Here is what I think is happening.
> 1. The job in question started an completed on 05/02/2015. So ideally the 
> history file location should be /mapred/history/done/2015/05/02/{000002}/
> 2. But instead HistoryFileManager looks at directory 
> /mapred/history/done/2015/04/02/{000002}/ and fails
> Looking at the logic in 
> {code}org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanOldDirsForJob(JobId)
>  {code}
>  of how the idtoDateString cache is created looks like the key is independent 
> of the RM start  time,
> So if you had 2 jobs job_RMstarttime1_0001 and job_RMstarttime2_0001, the 
> idtoDateString cache will have the following entries
> 000000001  ->  { job_RMstarttime1_0001historydir, 
> job_RMstarttime2_0001historyDir).
> 3. If job_RMstarttime1_0001 is older than "mapreduce.jobhistory.max-age-ms" 
> we delete the history info from HDFS.
> 4. For job_RMstarttime2_0001historyDir when we try and query it fails with a 
> filenotFoundException.
> Either the keys should be aware of RM starttime or before  
> HistoryFileManager.scanDirectory does a list status it should check if the 
> path exists to avoid file not found exception.
> {code}
> private static List<FileStatus> scanDirectory(Path path, FileContext fc,
>       PathFilter pathFilter) throws IOException {
>     path = fc.makeQualified(path);
>     List<FileStatus> jhStatusList = new ArrayList<FileStatus>();
>     if(!fc.exists(path)) {
>          return jhStatusList
>     }
>     RemoteIterator<FileStatus> fileStatusIter = fc.listStatus(path);
>     while (fileStatusIter.hasNext()) {
>       FileStatus fileStatus = fileStatusIter.next();
>       Path filePath = fileStatus.getPath();
>       if (fileStatus.isFile() && pathFilter.accept(filePath)) {
>         jhStatusList.add(fileStatus);
>       }
>     }
>     return jhStatusList;
>   }
> {code}
> Complete stack trace: 
> gslog`20150504141445.816``263424`0`0````````1892499996-10858515`754671855`/ex/UnhandledException.jsp`JAVA.FileNotFoundException
>  -> java.io.IOException: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.YarnRuntimeException):
>  java.io.FileNotFoundException: File /mapred/history/done/2015/04/02/000002 
> does not exist.
>       at 
> org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:147)
>       at 
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:217)
>       at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:203)
>       at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:199)
>       at java.security.AccessController.doPrivileged(Native Method)
>       at javax.security.auth.Subject.doAs(Subject.java:415)
>       at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>       at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:199)
>       at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getJobReport(HistoryClientService.java:231)
>       at 
> org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getJobReport(MRClientProtocolPBServiceImpl.java:122)
>       at 
> org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:275)
>       at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
>       at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
>       at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
>       at java.security.AccessController.doPrivileged(Native Method)
>       at javax.security.auth.Subject.doAs(Subject.java:415)
>       at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
> Caused by: java.io.FileNotFoundException: File 
> /mapred/history/done/2015/04/02/000002 does not exist.
>       at org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:205)
>       at org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:189)
>       at org.apache.hadoop.fs.Hdfs$2.<init>(Hdfs.java:171)
>       at org.apache.hadoop.fs.Hdfs.listStatusIterator(Hdfs.java:171)
>       at org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1392)
>       at org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1387)
>       at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
>       at org.apache.hadoop.fs.FileContext.listStatus(FileContext.java:1387)
>       at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanDirectory(HistoryFileManager.java:739)
>       at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanDirectoryForHistoryFiles(HistoryFileManager.java:752)
>       at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanOldDirsForJob(HistoryFileManager.java:909)
>       at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.getFileInfo(HistoryFileManager.java:938)
>       at 
> org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:132)
>       ... 18 more
>  at 
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:334)`hadoop.client.NetworkedHadoopJobClientImpl`getCounters`RATEOF-BLOCK
>  Date: Mon May 04 14:14:45 GMT 2015
>  Hostname: gs0-app2-1-chi.ops.sfdc.net
>  SettingsPath: prod.-.chi.gs0.app
>  JavaVersion: 1.7.0_76
>  UniqueId: 1892499996-10858515
>  StackTraceId: 754671855
>  Allowed: 865202
>  Blocked: 3
>  ThreadId: 263424
>  ThreadName: /ex/UnhandledException.jsp
>  Category: JAVA.FileNotFoundException
>  SubCategory: BaseSfdcGack
>  SourceClassName: hadoop.client.NetworkedHadoopJobClientImpl
>  SourceMethodName: getCounters
>  OrganizationId: null
>  UserId: null
>  Subject: java.io.IOException: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.YarnRuntimeException):
>  java.io.FileNotFoundException: File /mapred/history/done/2015/04/02/000002 
> does not exist.
>               at 
> org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:147)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:217)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:203)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:199)
>               at java.security.AccessController.doPrivileged(Native Method)
>               at javax.security.auth.Subject.doAs(Subject.java:415)
>               at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:199)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getJobReport(HistoryClientService.java:231)
>               at 
> org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getJobReport(MRClientProtocolPBServiceImpl.java:122)
>               at 
> org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:275)
>               at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>               at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
>               at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
>               at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
>               at java.security.AccessController.doPrivileged(Native Method)
>               at javax.security.auth.Subject.doAs(Subject.java:415)
>               at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>               at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
>       Caused by: java.io.FileNotFoundException: File 
> /mapred/history/done/2015/04/02/000002 does not exist.
>               at 
> org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:205)
>               at 
> org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:189)
>               at org.apache.hadoop.fs.Hdfs$2.<init>(Hdfs.java:171)
>               at org.apache.hadoop.fs.Hdfs.listStatusIterator(Hdfs.java:171)
>               at 
> org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1392)
>               at 
> org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1387)
>               at 
> org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
>               at 
> org.apache.hadoop.fs.FileContext.listStatus(FileContext.java:1387)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanDirectory(HistoryFileManager.java:739)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanDirectoryForHistoryFiles(HistoryFileManager.java:752)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanOldDirsForJob(HistoryFileManager.java:909)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.getFileInfo(HistoryFileManager.java:938)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:132)
>               ... 18 more
>  ExtendedMessage: null
>  Instance: null
>  Level: SEVERE
>  Thrown: hadoop.client.HadoopClientException: java.io.IOException: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.YarnRuntimeException):
>  java.io.FileNotFoundException: File /mapred/history/done/2015/04/02/000002 
> does not exist.
>               at 
> org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:147)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:217)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:203)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:199)
>               at java.security.AccessController.doPrivileged(Native Method)
>               at javax.security.auth.Subject.doAs(Subject.java:415)
>               at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:199)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getJobReport(HistoryClientService.java:231)
>               at 
> org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getJobReport(MRClientProtocolPBServiceImpl.java:122)
>               at 
> org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:275)
>               at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>               at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
>               at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
>               at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
>               at java.security.AccessController.doPrivileged(Native Method)
>               at javax.security.auth.Subject.doAs(Subject.java:415)
>               at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>               at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
>       Caused by: java.io.FileNotFoundException: File 
> /mapred/history/done/2015/04/02/000002 does not exist.
>               at 
> org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:205)
>               at 
> org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:189)
>               at org.apache.hadoop.fs.Hdfs$2.<init>(Hdfs.java:171)
>               at org.apache.hadoop.fs.Hdfs.listStatusIterator(Hdfs.java:171)
>               at 
> org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1392)
>               at 
> org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1387)
>               at 
> org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
>               at 
> org.apache.hadoop.fs.FileContext.listStatus(FileContext.java:1387)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanDirectory(HistoryFileManager.java:739)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanDirectoryForHistoryFiles(HistoryFileManager.java:752)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanOldDirsForJob(HistoryFileManager.java:909)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.getFileInfo(HistoryFileManager.java:938)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:132)
>               ... 18 more
>   Cause0: java.io.IOException: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.YarnRuntimeException):
>  java.io.FileNotFoundException: File /mapred/history/done/2015/04/02/000002 
> does not exist.
>               at 
> org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:147)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:217)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:203)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:199)
>               at java.security.AccessController.doPrivileged(Native Method)
>               at javax.security.auth.Subject.doAs(Subject.java:415)
>               at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:199)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getJobReport(HistoryClientService.java:231)
>               at 
> org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getJobReport(MRClientProtocolPBServiceImpl.java:122)
>               at 
> org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:275)
>               at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>               at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
>               at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
>               at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
>               at java.security.AccessController.doPrivileged(Native Method)
>               at javax.security.auth.Subject.doAs(Subject.java:415)
>               at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>               at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
>       Caused by: java.io.FileNotFoundException: File 
> /mapred/history/done/2015/04/02/000002 does not exist.
>               at 
> org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:205)
>               at 
> org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:189)
>               at org.apache.hadoop.fs.Hdfs$2.<init>(Hdfs.java:171)
>               at org.apache.hadoop.fs.Hdfs.listStatusIterator(Hdfs.java:171)
>               at 
> org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1392)
>               at 
> org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1387)
>               at 
> org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
>               at 
> org.apache.hadoop.fs.FileContext.listStatus(FileContext.java:1387)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanDirectory(HistoryFileManager.java:739)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanDirectoryForHistoryFiles(HistoryFileManager.java:752)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanOldDirsForJob(HistoryFileManager.java:909)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.getFileInfo(HistoryFileManager.java:938)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:132)
>               ... 18 more
>  Cause0-StackTrace: 
>       at 
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:334)
>       at 
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419)
>       at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:532)
>       at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:183)
>       at 
> hadoop.client.NetworkedHadoopJobClientImpl.getJob(NetworkedHadoopJobClientImpl.java:133)
>       at 
> hadoop.client.NetworkedHadoopJobClientImpl.getCounters(NetworkedHadoopJobClientImpl.java:112)
>       ... 110 shared with parent
>  Cause1: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.YarnRuntimeException):
>  java.io.FileNotFoundException: File /mapred/history/done/2015/04/02/000002 
> does not exist.
>               at 
> org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:147)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:217)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:203)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:199)
>               at java.security.AccessController.doPrivileged(Native Method)
>               at javax.security.auth.Subject.doAs(Subject.java:415)
>               at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:199)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getJobReport(HistoryClientService.java:231)
>               at 
> org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getJobReport(MRClientProtocolPBServiceImpl.java:122)
>               at 
> org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:275)
>               at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>               at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
>               at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
>               at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
>               at java.security.AccessController.doPrivileged(Native Method)
>               at javax.security.auth.Subject.doAs(Subject.java:415)
>               at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>               at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
>       Caused by: java.io.FileNotFoundException: File 
> /mapred/history/done/2015/04/02/000002 does not exist.
>               at 
> org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:205)
>               at 
> org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:189)
>               at org.apache.hadoop.fs.Hdfs$2.<init>(Hdfs.java:171)
>               at org.apache.hadoop.fs.Hdfs.listStatusIterator(Hdfs.java:171)
>               at 
> org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1392)
>               at 
> org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1387)
>               at 
> org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
>               at 
> org.apache.hadoop.fs.FileContext.listStatus(FileContext.java:1387)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanDirectory(HistoryFileManager.java:739)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanDirectoryForHistoryFiles(HistoryFileManager.java:752)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanOldDirsForJob(HistoryFileManager.java:909)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.getFileInfo(HistoryFileManager.java:938)
>               at 
> org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:132)
>               ... 18 more
>  Cause1-StackTrace: 
>       at org.apache.hadoop.ipc.Client.call(Client.java:1409)
>       at org.apache.hadoop.ipc.Client.call(Client.java:1362)
>       at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>       at com.sun.proxy.$Proxy328.getJobReport(Unknown Source)
>       at 
> org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:133)
>       at sun.reflect.GeneratedMethodAccessor4114.invoke(Unknown Source)
>       at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>       at java.lang.reflect.Method.invoke(Method.java:606)
>       at 
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:320)
>       ... 115 shared with parent Option-RacNode: 2
>   RequestParameters: 
>       jobStatus=SUCCEEDED
>       assignedHadoopJobId=job_1428500471449_2445



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to