pkumarsinha commented on pull request #3463:
URL: https://github.com/apache/hadoop/pull/3463#issuecomment-923030940
> 1. Is the current behaviour actually causing a problem?
>
> 2. Have you looked at all the implementations of fs.getXAttrs() and
verified that they always throw UnsupportedOperationException if unsupported.
>
>
> I can see from a purism perspective this is the correct thing to do. But I
worry that changing it now will break things.
>
> After all, if an IOE is raised here for some reason (network, auth etc)
distcp will go on to fail shortly afterwards
Thanks for the review @steveloughran !!
1) Yes, experienced recently this exception:
021-08-02 10:05:17,150 ERROR org.apache.hadoop.tools.DistCp:
[Thread-1973153]: XAttrs not supported on at least one file system:
org.apache.hadoop.tools.CopyListing$XAttrsNotSupportedException: XAttrs not
supported for file system: hdfs://someNN_host.tests.cloudera.com:8020
at
org.apache.hadoop.tools.util.DistCpUtils.checkFileSystemXAttrSupport(DistCpUtils.java:513)
~[hadoop-distcp-3.1.1.7.1.7.0-392.jar:?]
at
org.apache.hadoop.tools.CopyListing.validateFinalListing(CopyListing.java:202)
~[hadoop-distcp-3.1.1.7.1.7.0-392.jar:?]
at
org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:93)
~[hadoop-distcp-3.1.1.7.1.7.0-392.jar:?]
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:379)
~[hadoop-distcp-3.1.1.7.1.7.0-392.jar:?]
at org.apache.hadoop.tools.DistCp.prepareFileListing(DistCp.java:96)
~[hadoop-distcp-3.1.1.7.1.7.0-392.jar:?]
at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:216)
~[hadoop-distcp-3.1.1.7.1.7.0-392.jar:?]
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:193)
~[hadoop-distcp-3.1.1.7.1.7.0-392.jar:?]
at org.apache.hadoop.tools.DistCp.run(DistCp.java:155)
[hadoop-distcp-3.1.1.7.1.7.0-392.jar:?]
Didn't expect this call to fail for hdfs. And the only reason I could think
of is the improper exception handling. It could probably be an intermittent
network issue or so in this case.
2) The default behavior in FileSystem.java is to throw
UnsupportedOperationException. Couldn't notice any override where it is
silently ignored.
/**
* Get all of the xattr name/value pairs for a file or directory.
* Only those xattrs which the logged-in user has permissions to view
* are returned.
* <p>
* Refer to the HDFS extended attributes user documentation for details.
*
* @param path Path to get extended attributes
* @return Map describing the XAttrs of the file or directory
* @throws IOException IO failure
* @throws UnsupportedOperationException if the operation is unsupported
* (default outcome).
*/
public Map<String, byte[]> getXAttrs(Path path) throws IOException {
throw new UnsupportedOperationException(getClass().getSimpleName()
+ " doesn't support getXAttrs");
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
