Brutal Friday night. Coding < pussy. :)
On Fri, Oct 28, 2011 at 8:43 PM, Alex Gauthier <[email protected]>wrote: > > > On Fri, Oct 28, 2011 at 8:41 PM, Tom Melendez <[email protected]> wrote: > >> Hi Jay, >> >> Are you able to look at the logs or the web interface? Can you find >> out why it's getting killed? >> >> Also, can you verify that these ports are open and a process is >> connected to them (maybe with netstat)? >> >> http://www.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/ >> >> Thanks, >> >> Tom >> >> On Fri, Oct 28, 2011 at 7:57 PM, Jay Vyas <[email protected]> wrote: >> > Thanks tom : Thats interesting.... >> > >> > First, I tried, and it complained that the input directory didnt exist, >> so I >> > ran >> > $> hadoop fs -mkdir /user/cloudera/input >> > >> > Then, I tried to do this : >> > >> > $> hadoop jar /usr/lib/hadoop-0.20/hadoop-examples.jar grep input >> output2 >> > 'dfs[a-z.]+' >> > >> > And it seemed to start working ...... But then it abruptly printed >> "killed" >> > somehow at the end of the job [scroll down] ? >> > >> > Maybe this is related to why i cant connect ..... ?! >> > >> > 1) the hadoop jar 11/10/14 21:34:43 WARN util.NativeCodeLoader: Unable >> to >> > load native-hadoop library for your platform... using builtin-java >> classes >> > where applicable >> > 11/10/14 21:34:43 WARN snappy.LoadSnappy: Snappy native library not >> loaded >> > 11/10/14 21:34:43 INFO mapred.FileInputFormat: Total input paths to >> process >> > : 0 >> > 11/10/14 21:34:44 INFO mapred.JobClient: Running job: >> job_201110142010_0009 >> > 11/10/14 21:34:45 INFO mapred.JobClient: map 0% reduce 0% >> > 11/10/14 21:34:55 INFO mapred.JobClient: map 0% reduce 100% >> > 11/10/14 21:34:57 INFO mapred.JobClient: Job complete: >> job_201110142010_0009 >> > 11/10/14 21:34:57 INFO mapred.JobClient: Counters: 14 >> > 11/10/14 21:34:57 INFO mapred.JobClient: Job Counters >> > 11/10/14 21:34:57 INFO mapred.JobClient: Launched reduce tasks=1 >> > 11/10/14 21:34:57 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=5627 >> > 11/10/14 21:34:57 INFO mapred.JobClient: Total time spent by all >> reduces >> > waiting after reserving slots (ms)=0 >> > 11/10/14 21:34:57 INFO mapred.JobClient: Total time spent by all >> maps >> > waiting after reserving slots (ms)=0 >> > 11/10/14 21:34:57 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=5050 >> > 11/10/14 21:34:57 INFO mapred.JobClient: FileSystemCounters >> > 11/10/14 21:34:57 INFO mapred.JobClient: FILE_BYTES_WRITTEN=53452 >> > 11/10/14 21:34:57 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=86 >> > 11/10/14 21:34:57 INFO mapred.JobClient: Map-Reduce Framework >> > 11/10/14 21:34:57 INFO mapred.JobClient: Reduce input groups=0 >> > 11/10/14 21:34:57 INFO mapred.JobClient: Combine output records=0 >> > 11/10/14 21:34:57 INFO mapred.JobClient: Reduce shuffle bytes=0 >> > 11/10/14 21:34:57 INFO mapred.JobClient: Reduce output records=0 >> > 11/10/14 21:34:57 INFO mapred.JobClient: Spilled Records=0 >> > 11/10/14 21:34:57 INFO mapred.JobClient: Combine input records=0 >> > 11/10/14 21:34:57 INFO mapred.JobClient: Reduce input records=0 >> > 11/10/14 21:34:57 WARN mapred.JobClient: Use GenericOptionsParser for >> > parsing the arguments. Applications should implement Tool for the same. >> > 11/10/14 21:34:58 INFO mapred.FileInputFormat: Total input paths to >> process >> > : 1 >> > 11/10/14 21:34:58 INFO mapred.JobClient: Running job: >> job_201110142010_0010 >> > 11/10/14 21:34:59 INFO mapred.JobClient: map 0% reduce 0% >> > Killed >> > >> > >> > On Fri, Oct 28, 2011 at 8:24 PM, Tom Melendez <[email protected]> wrote: >> > >> >> Hi Jay, >> >> >> >> Some questions for you: >> >> >> >> - Does the hadoop client itself work from that same machine? >> >> - Are you actually able to run the hadoop example jar (in other words, >> >> your setup is valid otherwise)? >> >> - Is port 8020 actually available? (you can telnet or nc to it?) >> >> - What does jps show on the namenode? >> >> >> >> Thanks, >> >> >> >> Tom >> >> >> >> On Fri, Oct 28, 2011 at 4:04 PM, Jay Vyas <[email protected]> >> wrote: >> >> > Hi guys : Made more progress debugging my hadoop connection, but >> still >> >> > haven't got it working...... It looks like my VM (cloudera hadoop) >> won't >> >> > let me in. I find that there is no issue connecting to the name node >> - >> >> that >> >> > is , using hftp and 50070...... >> >> > >> >> > via standard HFTP as in here : >> >> > >> >> > //This method works fine - connecting directly to hadoop's namenode >> and >> >> > querying the filesystem >> >> > public static void main1(String[] args) throws Exception >> >> > { >> >> > String uri = "hftp://155.37.101.76:50070/";; >> >> > >> >> > System.out.println( "uri: " + uri ); >> >> > Configuration conf = new Configuration(); >> >> > >> >> > FileSystem fs = FileSystem.get( URI.create( uri ), conf ); >> >> > fs.printStatistics(); >> >> > } >> >> > >> >> > >> >> > But unfortunately, I can't get into hdfs ..... Any thoughts on this ? >> I >> >> am >> >> > modifying the uri to access port 8020 >> >> > which is what is in my core-site.xml . >> >> > >> >> > // This fails, resulting (trys to connect over and over again, >> >> eventually >> >> > gives up printing "already tried to connect 20 times"....) >> >> > public static void main(String[] args) >> >> > { >> >> > try { >> >> > String uri = "hdfs://155.37.101.76:8020/"; >> >> > >> >> > System.out.println( "uri: " + uri ); >> >> > Configuration conf = new Configuration(); >> >> > >> >> > FileSystem fs = FileSystem.get( URI.create( uri ), conf ); >> >> > fs.printStatistics(); >> >> > } catch (Exception e) { >> >> > // TODO Auto-generated catch block >> >> > e.printStackTrace(); >> >> > } >> >> > } >> >> > >> >> > The error message is : >> >> > >> >> > 11/10/28 19:03:38 INFO ipc.Client: Retrying connect to server: / >> >> > 155.37.101.76:8020. Already tried 0 time(s). >> >> > 11/10/28 19:03:39 INFO ipc.Client: Retrying connect to server: / >> >> > 155.37.101.76:8020. Already tried 1 time(s). >> >> > 11/10/28 19:03:40 INFO ipc.Client: Retrying connect to server: / >> >> > 155.37.101.76:8020. Already tried 2 time(s). >> >> > 11/10/28 19:03:41 INFO ipc.Client: Retrying connect to server: / >> >> > 155.37.101.76:8020. Already tried 3 time(s). >> >> > >> >> > Any thoughts on this would be *really* be appreciated ... Thanks >> guys. >> >> > >> >> >> > >> > >> > >> > -- >> > Jay Vyas >> > MMSB/UCHC >> > >> > >
