[
https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854237#action_12854237
]
Koji Noguchi commented on HDFS-1085:
------------------------------------
On the source datanode side, there was a log with
{quote}
2010-04-06 17:56:42,994 ERROR org.mortbay.log: /streamFile
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
at
org.mortbay.io.nio.SelectChannelEndPoint.updateKey(SelectChannelEndPoint.java:324)
at
org.mortbay.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:278)
at
org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:542)
at
org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569)
at
org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:946)
at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:646)
at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:577)
at
org.apache.hadoop.hdfs.server.namenode.StreamFile.doGet(StreamFile.java:68)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1124)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:663)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:361)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
2010-04-06 17:56:43,007 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/99.99.99.99:50010, dest: /88.88.88.88:49774, bytes: 61292544, op: HDFS_READ,
cliID: DFSClient_-563231950, offset: 0, srvID:
DS-1411469763-98.136.245.125-50010-1268765357890, blockid:
blk_-8819444732305373616_6553266, duration: 95558730000
{quote}
It was a distcp, so it should have read the whole 1 block(128MB) instead of
bytes: 61292544.
{quote}
1. blk_-8819444732305373616_6553266 len=134217728
{quote}
Nicholas pointed out to me that this is very similar to error observed in
HDFS-1024.
For distcp, thanks to the extra size checking from HADOOP-3294 we are ok, but
does this mean I cannot trust the result of hadoop '-get' through hftp ?
{quote}
% hadoop dfs -get hftp://somehost/somepath/file1 .
{quote}
I have a feeling that this command can *silently* fail with incomplete output.
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Koji Noguchi
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL
> 2010/0/part-00032 : java.io.IOException: File size not matched: copied
> 193855488 bytes (184.9m) to tmpfile
> (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from
> hftp://someotherhost/somepath/part-00032
> at
> org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow
> smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
