zhihai xu commented on YARN-2893:

I feel I know what cause this issue.
This issue most likely in the following code:
   Credentials credentials = new Credentials();
    DataInputByteBuffer dibb = new DataInputByteBuffer();
    if (container.getTokens() != null) {
      // TODO: Don't do this kind of checks everywhere.
we didn't rewind the token after credentials.readTokenStorageStream(dibb),
I checked the code in DataInputByteBuffer. It will move the position of 
ByteBuffer (container.getTokens) in DataInputByteBuffer.read.
(HeapByteBuffer is used for container.getTokens).
    public int read(byte[] b, int off, int len) {
      if (bidx >= buffers.length) {
        return -1;
      int cur = 0;
      do {
        int rem = Math.min(len, buffers[bidx].remaining());
        buffers[bidx].get(b, off, rem);
        cur += rem;
        off += rem;
        len -= rem;
      } while (len > 0 && ++bidx < buffers.length);
      pos += cur;
      return cur;
So If exception happen in AMLauncher.setupTokens before the ByteBuffer changed 
in container.setTokens.
Then the position of ByteBuffer of Tokens will be at the end and we will see 
this issue next time when we retry.
So it think it will be good to add  container.getTokens().rewind() after 
I will create a patch for this.

> AMLaucher: sporadic job failures due to EOFException in readTokenStorageStream
> ------------------------------------------------------------------------------
>                 Key: YARN-2893
>                 URL: https://issues.apache.org/jira/browse/YARN-2893
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.4.0
>            Reporter: Gera Shegalov
>            Assignee: zhihai xu
> MapReduce jobs on our clusters experience sporadic failures due to corrupt 
> tokens in the AM launch context.

This message was sent by Atlassian JIRA

Reply via email to