Steven Rand created HDFS-15191:
----------------------------------
Summary: EOF when reading legacy buffer in BlockTokenIdentifier
Key: HDFS-15191
URL: https://issues.apache.org/jira/browse/HDFS-15191
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs
Affects Versions: 3.2.1
Reporter: Steven Rand
We have an HDFS client application which recently upgraded from 3.2.0 to 3.2.1.
After this upgrade (but not before), we sometimes see these errors when this
application is used with clusters still running Hadoop 2.x (more specifically
CDH 5.12.1):
{code}
WARN [2020-02-24T00:54:32.856Z]
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory: I/O error constructing
remote block reader. (_sampled: true)
java.io.EOFException:
at java.io.DataInputStream.readByte(DataInputStream.java:272)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329)
at
org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.readFieldsLegacy(BlockTokenIdentifier.java:240)
at
org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.readFields(BlockTokenIdentifier.java:221)
at
org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:200)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:530)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:342)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:276)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:245)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:227)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.peerSend(SaslDataTransferClient.java:170)
at
org.apache.hadoop.hdfs.DFSUtilClient.peerFromSocketAndKey(DFSUtilClient.java:730)
at
org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2942)
at
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:822)
at
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:747)
at
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:380)
at
org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:644)
at
org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:575)
at
org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:757)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:829)
at java.io.DataInputStream.read(DataInputStream.java:100)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2314)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:2270)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2291)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:2246)
at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:765)
{code}
We get this warning for all DataNodes with a copy of the block, so the read
fails.
I haven't been able to figure out what changed between 3.2.0 and 3.2.1 to cause
this, but HDFS-13617 and HDFS-14611 seem related, so tagging [~vagarychen] in
case you have any ideas.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
