Can you retrieve log for appattempt_1440495451668_0258_000001 and see if there is some clue there ?
Cheers On Sun, Sep 13, 2015 at 3:28 AM, Bin Wang <wbi...@gmail.com> wrote: > There is some error logs in the executor and I don't know if it is related: > > 15/09/11 10:54:05 WARN ipc.Client: Exception encountered while connecting > to the server : > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$Inv > alidToken): Invalid AMRMToken from appattempt_1440495451668_0258_000001 > 15/09/11 10:54:05 WARN yarn.ApplicationMaster: Reporter thread fails 4 > time(s) in a row. > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid > AMRMToken from appattempt_1440495451668_0258_000001 > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79) > at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy22.allocate(Unknown Source) > at > org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:278) > at > org.apache.spark.deploy.yarn.YarnAllocator.allocateResources(YarnAllocator.scala:174) > at > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$1.run(ApplicationMaster.scala:323) > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1440495451668_0258_000001 > at org.apache.hadoop.ipc.Client.call(Client.java:1468) > at org.apache.hadoop.ipc.Client.call(Client.java:1399) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) > at com.sun.proxy.$Proxy21.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > ... 9 more > > ... > > 15/09/11 10:54:10 WARN ipc.Client: Exception encountered while connecting > to the server : > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$Inv > alidToken): Invalid AMRMToken from appattempt_1440495451668_0258_000001 > 15/09/11 10:54:10 INFO yarn.ApplicationMaster: Final app status: FAILED, > exitCode: 12, (reason: Exception was thrown 5 time(s) from Reporter thread.) > 15/09/11 10:54:10 INFO streaming.StreamingContext: Invoking > stop(stopGracefully=false) from shutdown hook > 15/09/11 10:54:10 INFO scheduler.ReceiverTracker: Sent stop signal to all > 1 receivers > 15/09/11 10:54:12 ERROR scheduler.ReceiverTracker: Deregistered receiver > for stream 0: Stopped by driver > > Tathagata Das <t...@databricks.com>于2015年9月13日周日 下午4:05写道： > >> Maybe the driver got restarted. See the log4j logs of the driver before >> it restarted. >> >> On Thu, Sep 10, 2015 at 11:32 PM, Bin Wang <wbi...@gmail.com> wrote: >> >>> I'm using spark streaming 1.4.0 and have a DStream that have all the >>> data it received. But today the history data in the DStream seems to be >>> lost suddenly. And the application UI also lost the streaming process time >>> and all the related data. Could any give some hint to debug this? Thanks. >>> >>> >>> >>
