[ https://issues.apache.org/jira/browse/YARN-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddhi Mehta updated YARN-3578: ------------------------------- Summary: Accessing Counters after RM restart results in stale cache (fails with FileNotFoundException) (was: Accessing Counters after RM restart fails with FileNotFoundException) > Accessing Counters after RM restart results in stale cache (fails with > FileNotFoundException) > --------------------------------------------------------------------------------------------- > > Key: YARN-3578 > URL: https://issues.apache.org/jira/browse/YARN-3578 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn > Affects Versions: 2.3.0 > Reporter: Siddhi Mehta > > When the job client tries to access counters for a recently completed job. > Here is what I think is happening. > 1. The job in question started an completed on 05/02/2015. So ideally the > history file location should be /mapred/history/done/2015/05/02/{000002}/ > 2. But instead HistoryFileManager looks at directory > /mapred/history/done/2015/04/02/{000002}/ and fails > Looking at the logic in > {code}org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanOldDirsForJob(JobId) > {code} > of how the idtoDateString cache is created looks like the key is independent > of the RM start time, > So if you had 2 jobs job_RMstarttime1_0001 and job_RMstarttime2_0001, the > idtoDateString cache will have the following entries > 000000001 -> { job_RMstarttime1_0001historydir, > job_RMstarttime2_0001historyDir). > 3. If job_RMstarttime1_0001 is older than "mapreduce.jobhistory.max-age-ms" > we delete the history info from HDFS. > 4. For job_RMstarttime2_0001historyDir when we try and query it fails with a > filenotFoundException. > Either the keys should be aware of RM starttime or before > HistoryFileManager.scanDirectory does a list status it should check if the > path exists to avoid file not found exception. > {code} > private static List<FileStatus> scanDirectory(Path path, FileContext fc, > PathFilter pathFilter) throws IOException { > path = fc.makeQualified(path); > List<FileStatus> jhStatusList = new ArrayList<FileStatus>(); > if(!fc.exists(path)) { > return jhStatusList > } > RemoteIterator<FileStatus> fileStatusIter = fc.listStatus(path); > while (fileStatusIter.hasNext()) { > FileStatus fileStatus = fileStatusIter.next(); > Path filePath = fileStatus.getPath(); > if (fileStatus.isFile() && pathFilter.accept(filePath)) { > jhStatusList.add(fileStatus); > } > } > return jhStatusList; > } > {code} > Complete stack trace: > gslog`20150504141445.816``263424`0`0````````1892499996-10858515`754671855`/ex/UnhandledException.jsp`JAVA.FileNotFoundException > -> java.io.IOException: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.YarnRuntimeException): > java.io.FileNotFoundException: File /mapred/history/done/2015/04/02/000002 > does not exist. > at > org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:147) > at > org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:217) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:203) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:199) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:199) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getJobReport(HistoryClientService.java:231) > at > org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getJobReport(MRClientProtocolPBServiceImpl.java:122) > at > org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:275) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980) > Caused by: java.io.FileNotFoundException: File > /mapred/history/done/2015/04/02/000002 does not exist. > at org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:205) > at org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:189) > at org.apache.hadoop.fs.Hdfs$2.<init>(Hdfs.java:171) > at org.apache.hadoop.fs.Hdfs.listStatusIterator(Hdfs.java:171) > at org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1392) > at org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1387) > at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) > at org.apache.hadoop.fs.FileContext.listStatus(FileContext.java:1387) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanDirectory(HistoryFileManager.java:739) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanDirectoryForHistoryFiles(HistoryFileManager.java:752) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanOldDirsForJob(HistoryFileManager.java:909) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.getFileInfo(HistoryFileManager.java:938) > at > org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:132) > ... 18 more > at > org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:334)`hadoop.client.NetworkedHadoopJobClientImpl`getCounters`RATEOF-BLOCK > Date: Mon May 04 14:14:45 GMT 2015 > Hostname: gs0-app2-1-chi.ops.sfdc.net > SettingsPath: prod.-.chi.gs0.app > JavaVersion: 1.7.0_76 > UniqueId: 1892499996-10858515 > StackTraceId: 754671855 > Allowed: 865202 > Blocked: 3 > ThreadId: 263424 > ThreadName: /ex/UnhandledException.jsp > Category: JAVA.FileNotFoundException > SubCategory: BaseSfdcGack > SourceClassName: hadoop.client.NetworkedHadoopJobClientImpl > SourceMethodName: getCounters > OrganizationId: null > UserId: null > Subject: java.io.IOException: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.YarnRuntimeException): > java.io.FileNotFoundException: File /mapred/history/done/2015/04/02/000002 > does not exist. > at > org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:147) > at > org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:217) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:203) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:199) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:199) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getJobReport(HistoryClientService.java:231) > at > org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getJobReport(MRClientProtocolPBServiceImpl.java:122) > at > org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:275) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980) > Caused by: java.io.FileNotFoundException: File > /mapred/history/done/2015/04/02/000002 does not exist. > at > org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:205) > at > org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:189) > at org.apache.hadoop.fs.Hdfs$2.<init>(Hdfs.java:171) > at org.apache.hadoop.fs.Hdfs.listStatusIterator(Hdfs.java:171) > at > org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1392) > at > org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1387) > at > org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) > at > org.apache.hadoop.fs.FileContext.listStatus(FileContext.java:1387) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanDirectory(HistoryFileManager.java:739) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanDirectoryForHistoryFiles(HistoryFileManager.java:752) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanOldDirsForJob(HistoryFileManager.java:909) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.getFileInfo(HistoryFileManager.java:938) > at > org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:132) > ... 18 more > ExtendedMessage: null > Instance: null > Level: SEVERE > Thrown: hadoop.client.HadoopClientException: java.io.IOException: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.YarnRuntimeException): > java.io.FileNotFoundException: File /mapred/history/done/2015/04/02/000002 > does not exist. > at > org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:147) > at > org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:217) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:203) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:199) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:199) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getJobReport(HistoryClientService.java:231) > at > org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getJobReport(MRClientProtocolPBServiceImpl.java:122) > at > org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:275) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980) > Caused by: java.io.FileNotFoundException: File > /mapred/history/done/2015/04/02/000002 does not exist. > at > org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:205) > at > org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:189) > at org.apache.hadoop.fs.Hdfs$2.<init>(Hdfs.java:171) > at org.apache.hadoop.fs.Hdfs.listStatusIterator(Hdfs.java:171) > at > org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1392) > at > org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1387) > at > org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) > at > org.apache.hadoop.fs.FileContext.listStatus(FileContext.java:1387) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanDirectory(HistoryFileManager.java:739) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanDirectoryForHistoryFiles(HistoryFileManager.java:752) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanOldDirsForJob(HistoryFileManager.java:909) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.getFileInfo(HistoryFileManager.java:938) > at > org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:132) > ... 18 more > Cause0: java.io.IOException: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.YarnRuntimeException): > java.io.FileNotFoundException: File /mapred/history/done/2015/04/02/000002 > does not exist. > at > org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:147) > at > org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:217) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:203) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:199) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:199) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getJobReport(HistoryClientService.java:231) > at > org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getJobReport(MRClientProtocolPBServiceImpl.java:122) > at > org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:275) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980) > Caused by: java.io.FileNotFoundException: File > /mapred/history/done/2015/04/02/000002 does not exist. > at > org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:205) > at > org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:189) > at org.apache.hadoop.fs.Hdfs$2.<init>(Hdfs.java:171) > at org.apache.hadoop.fs.Hdfs.listStatusIterator(Hdfs.java:171) > at > org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1392) > at > org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1387) > at > org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) > at > org.apache.hadoop.fs.FileContext.listStatus(FileContext.java:1387) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanDirectory(HistoryFileManager.java:739) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanDirectoryForHistoryFiles(HistoryFileManager.java:752) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanOldDirsForJob(HistoryFileManager.java:909) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.getFileInfo(HistoryFileManager.java:938) > at > org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:132) > ... 18 more > Cause0-StackTrace: > at > org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:334) > at > org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419) > at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:532) > at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:183) > at > hadoop.client.NetworkedHadoopJobClientImpl.getJob(NetworkedHadoopJobClientImpl.java:133) > at > hadoop.client.NetworkedHadoopJobClientImpl.getCounters(NetworkedHadoopJobClientImpl.java:112) > ... 110 shared with parent > Cause1: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.YarnRuntimeException): > java.io.FileNotFoundException: File /mapred/history/done/2015/04/02/000002 > does not exist. > at > org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:147) > at > org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:217) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:203) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:199) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:199) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getJobReport(HistoryClientService.java:231) > at > org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getJobReport(MRClientProtocolPBServiceImpl.java:122) > at > org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:275) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980) > Caused by: java.io.FileNotFoundException: File > /mapred/history/done/2015/04/02/000002 does not exist. > at > org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:205) > at > org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:189) > at org.apache.hadoop.fs.Hdfs$2.<init>(Hdfs.java:171) > at org.apache.hadoop.fs.Hdfs.listStatusIterator(Hdfs.java:171) > at > org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1392) > at > org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1387) > at > org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) > at > org.apache.hadoop.fs.FileContext.listStatus(FileContext.java:1387) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanDirectory(HistoryFileManager.java:739) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanDirectoryForHistoryFiles(HistoryFileManager.java:752) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanOldDirsForJob(HistoryFileManager.java:909) > at > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.getFileInfo(HistoryFileManager.java:938) > at > org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:132) > ... 18 more > Cause1-StackTrace: > at org.apache.hadoop.ipc.Client.call(Client.java:1409) > at org.apache.hadoop.ipc.Client.call(Client.java:1362) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy328.getJobReport(Unknown Source) > at > org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:133) > at sun.reflect.GeneratedMethodAccessor4114.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:320) > ... 115 shared with parent Option-RacNode: 2 > RequestParameters: > jobStatus=SUCCEEDED > assignedHadoopJobId=job_1428500471449_2445 -- This message was sent by Atlassian JIRA (v6.3.4#6332)