Hey Yan, This looks great! I added a few requests to the JIRA, if you have time.
Cheers, Chris On 3/11/14 7:20 PM, "Yan Fang" <[email protected]> wrote: >Hi Chris, > >Has opened an issue >SAMZA-181<https://issues.apache.org/jira/browse/SAMZA-181>and also >uploaded the patch. Let me know if there is something wrong in my >tutorial. Thank you! > >Cheers, > >Fang, Yan >[email protected] >+1 (206) 849-4108 > > >On Tue, Mar 11, 2014 at 10:40 AM, ><[email protected]>wrote: > >> Thanks Chris, Yan, >> >> Let me try that. >> >> -----Original Message----- >> From: Chris Riccomini [mailto:[email protected]] >> Sent: Tuesday, March 11, 2014 10:22 AM >> To: [email protected] >> Subject: Re: Failed to package using mvn >> >> Hey Yan, >> >> Awesome!The location where you can add your .md is here: >> >> docs/learn/tutorials/0.7.0/ >> >> >> Here's a link to the code tree: >> >> >> >>https://git-wip-us.apache.org/repos/asf?p=incubator-samza.git;a=tree;f=do >>cs >> >>/learn/tutorials/0.7.0;h=ef117f4066f14a00f50f0f6fca17903130448312;hb=HEAD >> >> You can get the code here: >> >> git clone http://git-wip-us.apache.org/repos/asf/incubator-samza.git >> >> >> Once you write the .md, just throw it up on a JIRA, and one of us can >> merge it in. >> >> Re: hdfs-site.xml, ah ha, that's what I figured. This is good to know. >>So >> you just copy your hdfs-site.xml from your NodeManager's conf directory >> into your local hdfs-site.xml. >> >> Cheers, >> Chris >> >> On 3/11/14 10:16 AM, "Yan Fang" <[email protected]> wrote: >> >> >Hi Chris, >> > >> >Sure. I just do not know how/where to contribute this page...*_* >> > >> >Oh, I mean the same this as you mentioned in the *Cluster >> >Installation*thread: >> > >> >*"2. Get a copy of one of your NM's yarn-site.xml and put it somewhere >> >on* >> > >> >*your desktop (I usually use ~/.yarn/conf/yarn-site.xml). Note that >> >there'sa "conf" directory there. This is mandatory."* >> > >> >So I just copy the hdfs-site.xml to ~/.yarn/conf/hdfs-site.xml. Thank >>you. >> > >> >Cheers, >> > >> >Fang, Yan >> >[email protected] >> >+1 (206) 849-4108 >> > >> > >> >On Tue, Mar 11, 2014 at 10:10 AM, Chris Riccomini >> ><[email protected]>wrote: >> > >> >> Hey Yan, >> >> >> >> Would you be up for contributing a tutorial page that describes this? >> >>This >> >> is really useful information. Our docs are just simple .md files in >> >>the main code base. >> >> >> >> Regarding step (3), is the hdfs-site.xml put into the conf folder for >> >>the NM boxes, or on the client side (where run-job.sh is run)? >> >> >> >> Cheers, >> >> Chris >> >> >> >> On 3/11/14 10:07 AM, "Yan Fang" <[email protected]> wrote: >> >> >> >> >Hi Sonali, >> >> > >> >> >The way I make Samza run with HDFS is following: >> >> > >> >> >1. include hdfs jar in Samza jar tar.gz. >> >> >2. you may also want to make sure the hadoop-common.jar has the same >> >> >version as your hdfs jar. Otherwise, you may have configuration >> >> >error popping out. >> >> >3. then put hdfs-site.xml to conf folder, the same folder as the >> >> >yarn-site.xml 4. all other steps are not changed. >> >> > >> >> >Hope this will help. Thank you. >> >> > >> >> >Cheers, >> >> > >> >> >Fang, Yan >> >> >[email protected] >> >> >+1 (206) 849-4108 >> >> > >> >> > >> >> >On Tue, Mar 11, 2014 at 9:25 AM, Chris Riccomini >> >> ><[email protected]>wrote: >> >> > >> >> >> Hey Sonali, >> >> >> >> >> >> I believe that you need to make sure that the HDFS jar is in your >> >> >>.tar.gz file, as you've said. >> >> >> >> >> >> If that doesn't work, you might need to define this setting in >> >> >> core-site.xml on the machine you're running run-job.sh on: >> >> >> >> >> >> <property> >> >> >> <name>fs.hdfs.impl</name> >> >> >> <value>org.apache.hadoop.hdfs.DistributedFileSystem</value> >> >> >> <description>The FileSystem for hdfs: uris.</description> >> >> >> </property> >> >> >> >> >> >> >> >> >> You might also need to configure your NodeManagers to have the >> >> >> HDFS >> >>file >> >> >> system impl as well. >> >> >> >> >> >> I've never run Samza with HDFS, so I'm guessing here. Perhaps >> >> >>someone else on the list has been successful with this? >> >> >> >> >> >> Cheers, >> >> >> Chris >> >> >> >> >> >> On 3/10/14 3:59 PM, "[email protected]" >> >> >> <[email protected]> wrote: >> >> >> >> >> >> >Hello, >> >> >> > >> >> >> >I fixed this by starting from scratch with gradlew. But now when >> >> >> >I >> >>run >> >> >>my >> >> >> >job it throws this error: >> >> >> >Exception in thread "main" java.io.IOException: No FileSystem for >> >> >>scheme: >> >> >> >hdfs >> >> >> > at >> >> >> >> >> >> >>>>>org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java: >> >>>>>242 >> >>>>>1) >> >> >> > at >> >> >> >> >>>org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428 >> >>>) >> >> >> > at >> >> >>org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88) >> >> >> > at >> >> >> >> >>>org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:246 >> >>>7) >> >> >> > at >> >> >>org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449) >> >> >> > at >>org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367) >> >> >> > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287) >> >> >> > at >> >> >> >> >> >> >>>>>>>org.apache.samza.job.yarn.ClientHelper.submitApplication(ClientHelpe >>>>>>>r. >> >>>>>sc >> >> >>>al >> >> >> >a:111) >> >> >> > at >> >>org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:55) >> >> >> > at >> >>org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:48) >> >> >> > at >>org.apache.samza.job.JobRunner.run(JobRunner.scala:100) >> >> >> > at >>org.apache.samza.job.JobRunner$.main(JobRunner.scala:75) >> >> >> > at org.apache.samza.job.JobRunner.main(JobRunner.scala) >> >> >> > >> >> >> >I looked at the samza job tar.gz and it doesn't have a >> >> >> >Hadoop-hdfs >> >>jar. >> >> >> >Is that why I get this error? >> >> >> > >> >> >> >Thanks, >> >> >> >Sonali >> >> >> > >> >> >> >From: Parthasarathy, Sonali >> >> >> >Sent: Monday, March 10, 2014 11:25 AM >> >> >> >To: [email protected] >> >> >> >Subject: Failed to package using mvn >> >> >> > >> >> >> >Hi, >> >> >> > >> >> >> >When I tried to do a mvn clean package of my hello-samza project, >> >> >> >I >> >>get >> >> >> >the following error. Has anyone seen this before? >> >> >> > >> >> >> >[ERROR] Failed to execute goal on project samza-wikipedia: Could >> >> >> >not resolve dependencies for project >> samza:samza-wikipedia:jar:0.7.0: >> >>Could >> >> >> >not find artifact org.apache.samza:samza-kv_2.10:jar:0.7.0 in >> >> >> >apache-releases >> >>(https://repository.apache.org/content/groups/public) >> >> >>-> >> >> >> >[Help 1] >> >> >> >[ERROR] >> >> >> >[ERROR] To see the full stack trace of the errors, re-run Maven >> >> >> >with >> >> >>the >> >> >> >-e switch. >> >> >> >[ERROR] Re-run Maven using the -X switch to enable full debug >> >>logging. >> >> >> >[ERROR] >> >> >> >[ERROR] For more information about the errors and possible >> >>solutions, >> >> >> >please read the following articles: >> >> >> >[ERROR] [Help 1] >> >> >> > >> >> >> >> >> >> >> >> >> >>http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionE >> >>xce >> >> >>p >> >> >> >tion >> >> >> >[ERROR] >> >> >> >[ERROR] After correcting the problems, you can resume the build >> >> >> >with >> >> >>the >> >> >> >command >> >> >> >[ERROR] mvn <goals> -rf :samza-wikipedia >> >> >> > >> >> >> >Thanks, >> >> >> >Sonali >> >> >> > >> >> >> >Sonali Parthasarathy >> >> >> >R&D Developer, Data Insights >> >> >> >Accenture Technology Labs >> >> >> >703-341-7432 >> >> >> > >> >> >> > >> >> >> >________________________________ >> >> >> > >> >> >> >This message is for the designated recipient only and may contain >> >> >> >privileged, proprietary, or otherwise confidential information. >> >> >> >If >> >>you >> >> >> >have received it in error, please notify the sender immediately >> >> >> >and delete the original. Any other use of the e-mail by you is >> >>prohibited. >> >> >> >Where allowed by local law, electronic communications with >> >> >> >Accenture >> >> >>and >> >> >> >its affiliates, including e-mail and instant messaging (including >> >> >> >content), may be scanned by our systems for the purposes of >> >>information >> >> >> >security and assessment of internal compliance with Accenture >> >>policy. >> >> >> >> >> >> >>>>>___________________________________________________________________ >> >>>>>___ >> >>>>>__ >> >> >>>__ >> >> >> >____________ >> >> >> > >> >> >> >www.accenture.com >> >> >> >> >> >> >> >> >> >> >> >> >> >> ________________________________ >> >> This message is for the designated recipient only and may contain >> privileged, proprietary, or otherwise confidential information. If you >>have >> received it in error, please notify the sender immediately and delete >>the >> original. Any other use of the e-mail by you is prohibited. Where >>allowed >> by local law, electronic communications with Accenture and its >>affiliates, >> including e-mail and instant messaging (including content), may be >>scanned >> by our systems for the purposes of information security and assessment >>of >> internal compliance with Accenture policy. >> >> >>_________________________________________________________________________ >>_____________ >> >> www.accenture.com >> >>
