Hi Gabriel, Thanks for your quick reply. You are correct, I have mixed up the two modes, being in pseudo-distributed mode, I thought I was running the job in local mode, and tried to access the local file system. I've switched back to the standalone mode and now it's working.
Thanks & Regards, -Pavithra On Mon, Nov 3, 2014 at 3:16 PM, Gabriel Reid <[email protected]> wrote: > Hi Pavithra, > > This looks like there is probably a difference in where you're putting > your input files and where Hadoop is expecting to find them. > Are your input files on your local file system, or on HDFS? And is > Hadoop set up to work with your local file system or HDFS? > > The path "/home/pavithra/in" looks like it is likely a local path > (i.e. on the drive of your machine where you're running the command). > It could be that when hadoop is trying to read the files, it's > connecting to HDFS, and not your local file system. > > An easy way to check this is to type "hadoop dfs -ls > /home/pavithra/in". That command should list the input files you want > to use for your Crunch job, but I'm expecting that it will say that > that path does not exist (because it's looking on HDFS). Could you try > this out and then post back what you get? > > - Gabriel > > > On Mon, Nov 3, 2014 at 10:05 AM, Pavithra Siriwardena > <[email protected]> wrote: > > Hi folks, > > > > I am trying to run my first Crunch pipeline, using the example project > given > > in Crunch Home Page. I'm using Hadoop 2.51 and the Crunch0.11.0. > > > > When I try to run it locally with using Hadoop's launcher script, > > > > hadoop jar target/crunch-demo-1.0-SNAPSHOT-job.jar /home/pavithra/in > > /home/pavithra/out > > > > I get the error following error saying 'Input source Text() does not > > exist!'; Note that the input text files are in '/home/pavithra/in'. > > > > Exception in thread "main" java.lang.IllegalStateException: Input source > > Text(/home/pavithra/in) does not exist! > > at > > > org.apache.crunch.impl.dist.collect.BaseInputCollection.getSizeInternal(BaseInputCollection.java:66) > > at > > > org.apache.crunch.impl.dist.collect.PCollectionImpl.getSize(PCollectionImpl.java:315) > > at > > > org.apache.crunch.impl.dist.collect.BaseDoCollection.getSizeInternal(BaseDoCollection.java:49) > > at > > > org.apache.crunch.impl.dist.collect.PCollectionImpl.getSize(PCollectionImpl.java:315) > > at > > > org.apache.crunch.impl.dist.collect.BaseDoCollection.getSizeInternal(BaseDoCollection.java:49) > > at > > > org.apache.crunch.impl.dist.collect.PCollectionImpl.getSize(PCollectionImpl.java:315) > > at > > > org.apache.crunch.util.PartitionUtils.getRecommendedPartitions(PartitionUtils.java:51) > > at > > > org.apache.crunch.util.PartitionUtils.getRecommendedPartitions(PartitionUtils.java:40) > > at org.apache.crunch.lib.Aggregate.count(Aggregate.java:55) > > at > > > org.apache.crunch.impl.dist.collect.PCollectionImpl.count(PCollectionImpl.java:245) > > at com.example.WordCount.run(WordCount.java:54) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > > at com.example.WordCount.main(WordCount.java:22) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > > > > Can you please guide me on this problem... > > > > Thanks > > -Pavithra > > > > > -- -Pavithra
