[jira] [Updated] (HDFS-9455) In distcp, Invalid Argument Error thrown in case of filesystem operation failure
[
https://issues.apache.org/jira/browse/HDFS-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Archana T updated HDFS-9455:
Assignee: Daisuke Kobayashi (was: Archana T)
> In distcp, Invalid Argument Error thrown in case of filesystem operation
> failure
>
>
> Key: HDFS-9455
> URL: https://issues.apache.org/jira/browse/HDFS-9455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp, security
>Reporter: Archana T
>Assignee: Daisuke Kobayashi
>Priority: Minor
>
> When Filesystem Operation failure happens during discp,
> Wrong exception : Invalid Argument thrown along with distcp command Usage.
> {color:red}
> hadoop distcp webhdfs://IP:25003/test/testfile webhdfs://IP:25003/myp
> Invalid arguments: Unexpected end of file from server
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> -f List of files that need to be copied
> -filelimit (Deprecated!) Limit number of files copied
>to <= n
> -iIgnore failures during copy
> .
> {color}
> Instead Proper Exception has to be thrown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9455) In distcp, Invalid Argument Error thrown in case of filesystem operation failure
[
https://issues.apache.org/jira/browse/HDFS-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daisuke Kobayashi updated HDFS-9455:
Attachment: HDFS-9455.01.patch
Thanks! Uploaded a patch as HDFS-9455.01.patch. This catches SocketException
explicitly and log the following message when it fails to determine the target
path. I've confirmed this does work with upstream too.
{noformat}
[daisuke@test2 ~]$ hadoop distcp webhdfs://test2:50470/user/daisuke/hosts
webhdfs://test2:50470/user/daisuke/dir
16/01/08 18:42:09 WARN security.UserGroupInformation:
PriviledgedActionException as:daisuke@HADOOP (auth:KERBEROS)
cause:java.net.SocketException: Unexpected end of file from server
16/01/08 18:42:09 WARN security.UserGroupInformation:
PriviledgedActionException as:daisuke@HADOOP (auth:KERBEROS)
cause:java.net.SocketException: Unexpected end of file from server
16/01/08 18:42:09 ERROR tools.DistCp: An error occurs while getting target path:
java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
at
java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:336)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:614)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:464)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:493)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:489)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1307)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:239)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:429)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:450)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:697)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:609)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:464)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:493)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:489)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:844)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:859)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1409)
at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:229)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:118)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:458)
[daisuke@test2 ~]$
{noformat}
I am still unsure if returning {{DistCpConstants.UNKNOWN_ERROR}} is appropriate
though. Can you advice me, [~yzhangal]?
> In distcp, Invalid Argument Error thrown in case of filesystem operation
> failure
>
>
> Key: HDFS-9455
> URL: https://issues.apache.org/jira/browse/HDFS-9455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp, security
>Reporter: Archana T
>Assignee: Daisuke Kobayashi
>Priority: Minor
> Attachments: HDFS-9455.01.patch
>
>
> When Filesystem Operation failure happens during discp,
> Wrong exception : Invalid Argument thrown along with distcp command Usage.
> {color:red}
> hadoop distcp
[jira] [Updated] (HDFS-9455) In distcp, Invalid Argument Error thrown in case of filesystem operation failure
[
https://issues.apache.org/jira/browse/HDFS-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daisuke Kobayashi updated HDFS-9455:
Component/s: security
> In distcp, Invalid Argument Error thrown in case of filesystem operation
> failure
>
>
> Key: HDFS-9455
> URL: https://issues.apache.org/jira/browse/HDFS-9455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp, security
>Reporter: Archana T
>Assignee: Archana T
>Priority: Minor
>
> When Filesystem Operation failure happens during discp,
> Wrong exception : Invalid Argument thrown along with distcp command Usage.
> {color:red}
> hadoop distcp webhdfs://IP:25003/test/testfile webhdfs://IP:25003/myp
> Invalid arguments: Unexpected end of file from server
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> -f List of files that need to be copied
> -filelimit (Deprecated!) Limit number of files copied
>to <= n
> -iIgnore failures during copy
> .
> {color}
> Instead Proper Exception has to be thrown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9455) In distcp, Invalid Argument Error thrown in case of filesystem operation failure
[
https://issues.apache.org/jira/browse/HDFS-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Archana T updated HDFS-9455:
Summary: In distcp, Invalid Argument Error thrown in case of filesystem
operation failure (was: Invalid Argument Error thrown in case of filesystem
operation failure)
> In distcp, Invalid Argument Error thrown in case of filesystem operation
> failure
>
>
> Key: HDFS-9455
> URL: https://issues.apache.org/jira/browse/HDFS-9455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Reporter: Archana T
>Assignee: Archana T
>Priority: Minor
>
> When Filesystem Operation failure happens during discp,
> Wrong exception : Invalid Argument thrown along with distcp command Usage.
> {color:red}
> hadoop distcp webhdfs://IP:25003/test/testfile webhdfs://IP:25003/myp
> Invalid arguments: Unexpected end of file from server
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> -f List of files that need to be copied
> -filelimit (Deprecated!) Limit number of files copied
>to <= n
> -iIgnore failures during copy
> .
> {color}
> Instead Proper Exception has to be thrown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
