Dear Varun, thanks for the hints! Yep it's Mapreduce. Can't look at the code right now but I cannot remember overriding resolvePath. Will check it out and give feedback later on! Best Silvan
Von meinem iPhone gesendet > Am 08.05.2015 um 18:10 schrieb Varun Vasudev <[email protected]>: > > Hi Silvan, > > Are you trying to run MapReduce? You could look at > createApplicationSubmissionContext and createApplicationResource in > YarnRunner.java. Have you overridden the resolvePath function in your > filesystem implementation? > > -Varun > > From: Silvan Kaiser > Reply-To: "[email protected]" > Date: Friday, May 8, 2015 at 3:52 PM > To: "[email protected]" > Subject: URI missing scheme and authority in job start with new FileSystem > implementation > > > Hello! > I am running a new FileSystem implementation, testing it with a simple > standalone operation from the Hadoop "Single Node Setup" page. Staging > example jobs works alright but upon scheduling the container management fails > because of an URI without scheme > (and authority). > > NodeManager log snippet: > ... > > 2015-05-08 15:18:15,331 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: > Got exception parsing jobSubmitDir/job.splitmetainfo and value resource { > port: -1 file: > "/tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo" > } size: 771 timestamp: 1431091092000 type: FILE visibility: APPLICATION > 2015-05-08 15:18:15,331 WARN > org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: > Failed to parse resource-request > java.net.URISyntaxException: Expected scheme name at index 0: > :///tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo > > ... > > > The path element of the URI is correct, it simply is missing it's scheme and > authority but i cannot find out why so far. > Please note that the URIs used during staging are correct and contain scheme > & authority correctly. This means the incomplete URI is created during job > creation. > > My guess so far: this is either because I am missing some configuration for > the Nodemanager (therefore he does not know about my FileSystem > implementation and throws the unknown scheme & authority out the window) or > that the FileSystem implementation has > an error / did not override a method that requires adoption. > > Can someone please give me a hint where the URIs used by the NodeManager are > constructed prior to scheduling? This would allow me to trace back where the > incomplete URI is coming from... > > Best regards > Silvan > > > -- > Quobyte GmbH > Boyenstr. 41 - 10115 Berlin-Mitte - Germany > +49-30-814 591 800 - www.quobyte.com > <http://www.quobyte.com/><http://www.quobyte.com/> > Amtsgericht Berlin-Charlottenburg, HRB 149012B > Management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender > > > > > > > > > > -- > Quobyte GmbH > Hardenbergplatz 2 - 10623 Berlin - Germany > +49-30-814 591 800 - www.quobyte.com <http://www.quobyte.com/>Amtsgericht > Berlin-Charlottenburg, HRB 149012B > management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender -- -- *Quobyte* GmbH Hardenbergplatz 2 - 10623 Berlin - Germany +49-30-814 591 800 - www.quobyte.com Amtsgericht Berlin-Charlottenburg, HRB 149012B management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
