maobaolong commented on pull request #862: URL: https://github.com/apache/hadoop-ozone/pull/862#issuecomment-620504150
@elek Thank you for your review. The https://issues.apache.org/jira/browse/HDDS-3482 given some of my main env. I use the following suite - hadoop-2.7.2 as yarn client - hadoop-2.7.2 as yarn service (RM,NM) - ozone-0.6.0-SNAPSHOT So that, the core-site.xml of my yarn client is the following: Notice: it is different from your `docker-config`, i see you use the higher version hadoop, instead, i use hadoop-2.7.2 ```xml <configuration> <property> <name>fs.defaultFS</name> <value>o3fs://mybucket.myvol/</value> </property> <property> <name>fs.o3fs.impl</name> <value>org.apache.hadoop.fs.ozone.BasicOzoneFileSystem</value> </property> <property> <name>fs.AbstractFileSystem.o3fs.impl</name> <value>org.apache.hadoop.fs.ozone.BasicOzFs</value> </property> </configuration> ``` I executed the blow command to submit the MR to Yarn cluster, but it is failed. ```bash bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar teragen 1000 /input13 ``` The blow is error log after execute the above command. ``` 2020-04-23 15:15:04,544 [main] INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/root/.staging/job_1587619434776_0007 org.apache.hadoop.fs.InvalidPathException: Invalid path name Wrong FS: o3fs://mybucket.myvol/tmp/hadoop-yarn/staging/root/.staging/job_1587619434776_0007, expected: o3fs://mybucket.myvol/ at org.apache.hadoop.fs.AbstractFileSystem.checkPath(AbstractFileSystem.java:390) at org.apache.hadoop.fs.AbstractFileSystem.resolvePath(AbstractFileSystem.java:466) at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:348) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:285) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308) at org.apache.hadoop.examples.terasort.TeraGen.run(TeraGen.java:301) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.examples.terasort.TeraGen.main(TeraGen.java:305) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71) at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) ``` After some debug work, i focus on the following method from the hadoop-2.7.2. ```java public void checkPath(Path path) { URI uri = path.toUri(); String thatScheme = uri.getScheme(); String thatAuthority = uri.getAuthority(); if (thatScheme == null) { if (thatAuthority == null) { if (path.isUriPathAbsolute()) { return; } throw new InvalidPathException("relative paths not allowed:" + path); } else { throw new InvalidPathException( "Path without scheme with non-null authority:" + path); } } String thisScheme = this.getUri().getScheme(); String thisHost = this.getUri().getHost(); String thatHost = uri.getHost(); // Schemes and hosts must match. // Allow for null Authority for file:/// if (!thisScheme.equalsIgnoreCase(thatScheme) || (thisHost != null && !thisHost.equalsIgnoreCase(thatHost)) || (thisHost == null && thatHost != null)) { throw new InvalidPathException("Wrong FS: " + path + ", expected: " + this.getUri()); } // Ports must match, unless this FS instance is using the default port, in // which case the port may be omitted from the given URI int thisPort = this.getUri().getPort(); int thatPort = uri.getPort(); if (thatPort == -1) { // -1 => defaultPort of Uri scheme thatPort = this.getUriDefaultPort(); } if (thisPort != thatPort) { throw new InvalidPathException("Wrong FS: " + path + ", expected: " + this.getUri()); } } ``` `thisPort` is -1, and `thatPort` is -1 start, but `thatPort` is assigned by the abstract method `getUriDefaultPort()`, it is impl by DelegateToFileSystem which ozfs and OzfsBasic extended. unfortunately, this method impl good before we merge the patch https://github.com/apache/hadoop/commit/dc065dd64ca3e101b0c0a7bcc7d7a067b77d6c82 And key point is, we are using 2.7.2, so the `getUriDefaultPort` return 0. I have to override this method in the `BasicOzFs`. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
