[jira] [Commented] (HDFS-9455) In distcp, Invalid Argument Error thrown in case of filesystem operation failure
[
https://issues.apache.org/jira/browse/HDFS-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15827272#comment-15827272
]
Lu Tao commented on HDFS-9455:
--
@Jakub Zeman, you have to use swebhdfs schema, eg. hadoop fs -ls
swebhdfs://bda01:50470/jakubzeman/, that may works for you.
> In distcp, Invalid Argument Error thrown in case of filesystem operation
> failure
>
>
> Key: HDFS-9455
> URL: https://issues.apache.org/jira/browse/HDFS-9455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp, security
>Reporter: Archana T
>Assignee: Daisuke Kobayashi
>Priority: Minor
> Attachments: HDFS-9455.01.patch
>
>
> When Filesystem Operation failure happens during discp,
> Wrong exception : Invalid Argument thrown along with distcp command Usage.
> {color:red}
> hadoop distcp webhdfs://IP:25003/test/testfile webhdfs://IP:25003/myp
> Invalid arguments: Unexpected end of file from server
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> -f List of files that need to be copied
> -filelimit (Deprecated!) Limit number of files copied
>to <= n
> -iIgnore failures during copy
> .
> {color}
> Instead Proper Exception has to be thrown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9455) In distcp, Invalid Argument Error thrown in case of filesystem operation failure
[
https://issues.apache.org/jira/browse/HDFS-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825977#comment-15825977
]
Jakub Zeman commented on HDFS-9455:
---
Hi all,
would it be possible to increase prioriy of this issue. I have the same problem
accessing HDFS using WebHDFS on secure (https enabled) cluster:
{code}hadoop fs -ls webhdfs://bda01:50470/jakubzeman/
17/01/17 12:24:29 WARN security.UserGroupInformation:
PriviledgedActionException as:jakubze...@test.realm.com (auth:KERBEROS)
cause:java.net.SocketException: bda01:50470: Unexpected end of file from server
17/01/17 12:24:29 WARN security.UserGroupInformation:
PriviledgedActionException as:jakubze...@test.realm.com (auth:KERBEROS)
cause:java.net.SocketException: bda01:50470: Unexpected end of file from server
ls: bda01:50470: Unexpected end of file from serve{code}
This command works well on non-secure cluster.
Is there any way how to bypass https?
Jakub
> In distcp, Invalid Argument Error thrown in case of filesystem operation
> failure
>
>
> Key: HDFS-9455
> URL: https://issues.apache.org/jira/browse/HDFS-9455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp, security
>Reporter: Archana T
>Assignee: Daisuke Kobayashi
>Priority: Minor
> Attachments: HDFS-9455.01.patch
>
>
> When Filesystem Operation failure happens during discp,
> Wrong exception : Invalid Argument thrown along with distcp command Usage.
> {color:red}
> hadoop distcp webhdfs://IP:25003/test/testfile webhdfs://IP:25003/myp
> Invalid arguments: Unexpected end of file from server
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> -f List of files that need to be copied
> -filelimit (Deprecated!) Limit number of files copied
>to <= n
> -iIgnore failures during copy
> .
> {color}
> Instead Proper Exception has to be thrown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9455) In distcp, Invalid Argument Error thrown in case of filesystem operation failure
[
https://issues.apache.org/jira/browse/HDFS-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602374#comment-15602374
]
Steve Loughran commented on HDFS-9455:
--
* Anything raising an IOE should be interpreted as "something went wrong over
the wire", or at least in the FS client (auth, etc).
* the existing catching of Throwable is too broad...that can include things
like OOM other things which are very serious runtimes. Very much not "invalid
argument"
I think the handlers could go more like
# IOE -> rethrow
# Exception -> say "error"
# Throwable -> report as serious failure
> In distcp, Invalid Argument Error thrown in case of filesystem operation
> failure
>
>
> Key: HDFS-9455
> URL: https://issues.apache.org/jira/browse/HDFS-9455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp, security
>Reporter: Archana T
>Assignee: Daisuke Kobayashi
>Priority: Minor
> Attachments: HDFS-9455.01.patch
>
>
> When Filesystem Operation failure happens during discp,
> Wrong exception : Invalid Argument thrown along with distcp command Usage.
> {color:red}
> hadoop distcp webhdfs://IP:25003/test/testfile webhdfs://IP:25003/myp
> Invalid arguments: Unexpected end of file from server
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> -f List of files that need to be copied
> -filelimit (Deprecated!) Limit number of files copied
>to <= n
> -iIgnore failures during copy
> .
> {color}
> Instead Proper Exception has to be thrown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9455) In distcp, Invalid Argument Error thrown in case of filesystem operation failure
[
https://issues.apache.org/jira/browse/HDFS-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088929#comment-15088929
]
Archana T commented on HDFS-9455:
-
Hi [~daisuke.kobayashi]
I agee with the above proposal.
Assigning this Jira to you.
> In distcp, Invalid Argument Error thrown in case of filesystem operation
> failure
>
>
> Key: HDFS-9455
> URL: https://issues.apache.org/jira/browse/HDFS-9455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp, security
>Reporter: Archana T
>Assignee: Archana T
>Priority: Minor
>
> When Filesystem Operation failure happens during discp,
> Wrong exception : Invalid Argument thrown along with distcp command Usage.
> {color:red}
> hadoop distcp webhdfs://IP:25003/test/testfile webhdfs://IP:25003/myp
> Invalid arguments: Unexpected end of file from server
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> -f List of files that need to be copied
> -filelimit (Deprecated!) Limit number of files copied
>to <= n
> -iIgnore failures during copy
> .
> {color}
> Instead Proper Exception has to be thrown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9455) In distcp, Invalid Argument Error thrown in case of filesystem operation failure
[
https://issues.apache.org/jira/browse/HDFS-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088906#comment-15088906
]
Daisuke Kobayashi commented on HDFS-9455:
-
Thanks Yongjun for correcting me! Indeed, distcp runs within a single cluster
per the description and I could reproduce the issue as follows. [~archanat], is
this what you're also hitting, correct?
{noformat}
[daisuke@test1 jars]$ hadoop distcp webhdfs://test1:50470/user/daisuke/hosts
webhdfs://test1:50470/user/daisuke/dir/
16/01/08 17:40:22 WARN security.UserGroupInformation:
PriviledgedActionException as:daisuke@HADOOP (auth:KERBEROS)
cause:java.net.SocketException: Unexpected end of file from server
16/01/08 17:40:22 WARN security.UserGroupInformation:
PriviledgedActionException as:daisuke@HADOOP (auth:KERBEROS)
cause:java.net.SocketException: Unexpected end of file from server
16/01/08 17:40:22 ERROR tools.DistCp: Invalid arguments:
java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
at
java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:336)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:614)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:464)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:493)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:489)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1307)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:239)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:429)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:450)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:697)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:609)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:464)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:493)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:489)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:844)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:859)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1409)
at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:200)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:112)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:429)
Invalid arguments: Unexpected end of file from server
usage: distcp OPTIONS [source_path...]
OPTIONS
-append Reuse existing data in target files and
append new data to them if possible
<...snip...>
{noformat}
So my proposal is catching SocketException and print more meaningful message
like {{An error occurs while getting target path}}
If this is agreeable, do you allow me to create a patch to fix this,
[~archanat]?
> In distcp, Invalid Argument Error thrown in case of filesystem operation
> failure
>
>
> Key: HDFS-9455
> URL: https://issues.apache.org/jira/browse/HDFS-9455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp, security
>
[jira] [Commented] (HDFS-9455) In distcp, Invalid Argument Error thrown in case of filesystem operation failure
[
https://issues.apache.org/jira/browse/HDFS-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082670#comment-15082670
]
Archana T commented on HDFS-9455:
-
Hi [~daisuke.kobayashi]
I got the issue in OpenSource 2.7.1
Scenario --
In a secured cluster with ssl enabled, I was using "webhdfs" instead of
"swebhdfs" in below cmd --
hadoop distcp webhdfs://IP:25003/test/testfile webhdfs://IP:25003/myp
Reference-
HDFS-9483
> In distcp, Invalid Argument Error thrown in case of filesystem operation
> failure
>
>
> Key: HDFS-9455
> URL: https://issues.apache.org/jira/browse/HDFS-9455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Reporter: Archana T
>Assignee: Archana T
>Priority: Minor
>
> When Filesystem Operation failure happens during discp,
> Wrong exception : Invalid Argument thrown along with distcp command Usage.
> {color:red}
> hadoop distcp webhdfs://IP:25003/test/testfile webhdfs://IP:25003/myp
> Invalid arguments: Unexpected end of file from server
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> -f List of files that need to be copied
> -filelimit (Deprecated!) Limit number of files copied
>to <= n
> -iIgnore failures during copy
> .
> {color}
> Instead Proper Exception has to be thrown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9455) In distcp, Invalid Argument Error thrown in case of filesystem operation failure
[
https://issues.apache.org/jira/browse/HDFS-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082774#comment-15082774
]
Daisuke Kobayashi commented on HDFS-9455:
-
I see, [~archanat]. When using normal webhdfs protocol over SSL/TLS enabled
cluster, it should catch {{SocketException}} and print a stack trace. Here's a
result of my quick test, for instance:
{noformat}
[daisuke@test1 ~]$ hadoop distcp /user/daisuke/hosts webhdfs://target:50470/
16/01/05 19:11:34 WARN security.UserGroupInformation:
PriviledgedActionException as:daisuke@HADOOP (auth:KERBEROS)
cause:java.net.SocketException: Unexpected end of file from server
16/01/05 19:11:34 WARN security.UserGroupInformation:
PriviledgedActionException as:daisuke@HADOOP (auth:KERBEROS)
cause:java.net.SocketException: Unexpected end of file from server
16/01/05 19:11:34 ERROR tools.DistCp: Log.error:
java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
at
java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:336)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:614)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:464)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:493)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:489)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1307)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:239)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:429)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:450)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:697)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:609)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:464)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:493)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:489)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:844)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:859)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1409)
at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:206)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:113)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:435)
LOG.error: Unexpected end of file from server
{noformat}
Does this make sense?
> In distcp, Invalid Argument Error thrown in case of filesystem operation
> failure
>
>
> Key: HDFS-9455
> URL: https://issues.apache.org/jira/browse/HDFS-9455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp, security
>Reporter: Archana T
>Assignee: Archana T
>Priority: Minor
>
> When Filesystem Operation failure happens during discp,
> Wrong exception : Invalid Argument thrown along with distcp command Usage.
> {color:red}
> hadoop distcp webhdfs://IP:25003/test/testfile webhdfs://IP:25003/myp
> Invalid arguments: Unexpected end of file from server
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append
[jira] [Commented] (HDFS-9455) In distcp, Invalid Argument Error thrown in case of filesystem operation failure
[
https://issues.apache.org/jira/browse/HDFS-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082781#comment-15082781
]
Daisuke Kobayashi commented on HDFS-9455:
-
[~yzhangal], can you double check this proposal?
> In distcp, Invalid Argument Error thrown in case of filesystem operation
> failure
>
>
> Key: HDFS-9455
> URL: https://issues.apache.org/jira/browse/HDFS-9455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp, security
>Reporter: Archana T
>Assignee: Archana T
>Priority: Minor
>
> When Filesystem Operation failure happens during discp,
> Wrong exception : Invalid Argument thrown along with distcp command Usage.
> {color:red}
> hadoop distcp webhdfs://IP:25003/test/testfile webhdfs://IP:25003/myp
> Invalid arguments: Unexpected end of file from server
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> -f List of files that need to be copied
> -filelimit (Deprecated!) Limit number of files copied
>to <= n
> -iIgnore failures during copy
> .
> {color}
> Instead Proper Exception has to be thrown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9455) In distcp, Invalid Argument Error thrown in case of filesystem operation failure
[
https://issues.apache.org/jira/browse/HDFS-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083400#comment-15083400
]
Yongjun Zhang commented on HDFS-9455:
-
Hi Guys,
Thanks for reporting the issue and the work here. [~daisuke.kobayashi], Per
Archana's description at
https://issues.apache.org/jira/browse/HDFS-9455?focusedCommentId=15082670=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15082670
The command line is "hadoop distcp webhdfs://IP:25003/test/testfile
webhdfs://IP:25003/myp" in a secured cluster with ssl enabled (both src and tgt
are the same cluster).
Would you please try it out in your env?
The "Invalid argument" msg is not very clear, when ssl is enabled, if we are
using webhdfs instead of swebhdfs, possibly we can improve the message to
indicate that.
Thanks.
> In distcp, Invalid Argument Error thrown in case of filesystem operation
> failure
>
>
> Key: HDFS-9455
> URL: https://issues.apache.org/jira/browse/HDFS-9455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp, security
>Reporter: Archana T
>Assignee: Archana T
>Priority: Minor
>
> When Filesystem Operation failure happens during discp,
> Wrong exception : Invalid Argument thrown along with distcp command Usage.
> {color:red}
> hadoop distcp webhdfs://IP:25003/test/testfile webhdfs://IP:25003/myp
> Invalid arguments: Unexpected end of file from server
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> -f List of files that need to be copied
> -filelimit (Deprecated!) Limit number of files copied
>to <= n
> -iIgnore failures during copy
> .
> {color}
> Instead Proper Exception has to be thrown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9455) In distcp, Invalid Argument Error thrown in case of filesystem operation failure
[
https://issues.apache.org/jira/browse/HDFS-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082528#comment-15082528
]
Daisuke Kobayashi commented on HDFS-9455:
-
Thanks. Hmm, with which scenario does it occur? I've just tried to reproduce
the issue with the scenarios, but couldn't...
{noformat}
- target NameNode to be down
- target NameNode to be safemode
- wrong hostname to target NameNode
- wrong path to target
{noformat}
In which version of Hadoop are you using, [~archanat]? Mine is as follows:
{noformat}
# hadoop version
Hadoop 2.6.0-cdh5.5.1
{noformat}
> In distcp, Invalid Argument Error thrown in case of filesystem operation
> failure
>
>
> Key: HDFS-9455
> URL: https://issues.apache.org/jira/browse/HDFS-9455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Reporter: Archana T
>Assignee: Archana T
>Priority: Minor
>
> When Filesystem Operation failure happens during discp,
> Wrong exception : Invalid Argument thrown along with distcp command Usage.
> {color:red}
> hadoop distcp webhdfs://IP:25003/test/testfile webhdfs://IP:25003/myp
> Invalid arguments: Unexpected end of file from server
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> -f List of files that need to be copied
> -filelimit (Deprecated!) Limit number of files copied
>to <= n
> -iIgnore failures during copy
> .
> {color}
> Instead Proper Exception has to be thrown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9455) In distcp, Invalid Argument Error thrown in case of filesystem operation failure
[
https://issues.apache.org/jira/browse/HDFS-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070801#comment-15070801
]
Surendra Singh Lilhore commented on HDFS-9455:
--
Thanks [~archanat] for reporting this issue..
[~daisuke.kobayashi] this improper message is coming because of code exist in
DistCp.run()
{code}
try {
inputOptions = (OptionsParser.parse(argv));
setTargetPathExists();
LOG.info("Input Options: " + inputOptions);
} catch (Throwable e) {
LOG.error("Invalid arguments: ", e);
System.err.println("Invalid arguments: " + e.getMessage());
OptionsParser.usage();
return DistCpConstants.INVALID_ARGUMENT;
}
{code}
Here distcp will parse the argument and "setTargetPathExists()" will check the
existence of target path. If distcp is not able to connect with target
filesystem then it will throw the exception , but that exception will log with
wrong message.
{{LOG.error("Invalid arguments: ", e);}}
> In distcp, Invalid Argument Error thrown in case of filesystem operation
> failure
>
>
> Key: HDFS-9455
> URL: https://issues.apache.org/jira/browse/HDFS-9455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Reporter: Archana T
>Assignee: Archana T
>Priority: Minor
>
> When Filesystem Operation failure happens during discp,
> Wrong exception : Invalid Argument thrown along with distcp command Usage.
> {color:red}
> hadoop distcp webhdfs://IP:25003/test/testfile webhdfs://IP:25003/myp
> Invalid arguments: Unexpected end of file from server
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> -f List of files that need to be copied
> -filelimit (Deprecated!) Limit number of files copied
>to <= n
> -iIgnore failures during copy
> .
> {color}
> Instead Proper Exception has to be thrown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9455) In distcp, Invalid Argument Error thrown in case of filesystem operation failure
[
https://issues.apache.org/jira/browse/HDFS-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070613#comment-15070613
]
Daisuke Kobayashi commented on HDFS-9455:
-
[~archanat], what do you actually mean by 'Filesystem Operation failure'? I'd
like to know what sort of issue leads the improper message.
> In distcp, Invalid Argument Error thrown in case of filesystem operation
> failure
>
>
> Key: HDFS-9455
> URL: https://issues.apache.org/jira/browse/HDFS-9455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Reporter: Archana T
>Assignee: Archana T
>Priority: Minor
>
> When Filesystem Operation failure happens during discp,
> Wrong exception : Invalid Argument thrown along with distcp command Usage.
> {color:red}
> hadoop distcp webhdfs://IP:25003/test/testfile webhdfs://IP:25003/myp
> Invalid arguments: Unexpected end of file from server
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> -f List of files that need to be copied
> -filelimit (Deprecated!) Limit number of files copied
>to <= n
> -iIgnore failures during copy
> .
> {color}
> Instead Proper Exception has to be thrown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
